Intercoder Agreement Research Articles

Can a zero-shot learning Large Language Model code complex interview data?

Abstract Background Psychological autopsy is essential to establish theories, explore trends and identify previously unexplored psychosocial risk factors in suicide research. However, qualitative research has been scrutinized for being prone to interpretation bias, problems with accuracy, challenges to reproducibility, and is very time- and cost intensive. The current study aimed to investigate if a Large Language Model can achieve sufficient agreement with a researcher in the deductive coding of interview data obtained in a psychological autopsy study of suicide, to be integrated with qualitative research procedures. Methods Data from 38 interviews was deductively coded by a researcher and a LLAMA 3 based language model. The model performance was evaluated in four increasingly difficult coding tasks, including binary classification and data summarization. Intercoder agreement scores were calculated using Cohen’s Kappa. Results The preliminary results showed that the LLM achieved substantial agreement with the human coders for the binary classification task (.78). The variability in performance was influenced by code definitions. The results of the quality of LLM interpretation and summarization will also be presented. Conclusions State-of-the-art LLM can be easily integrated into the qualitative analyses of psychological autopsy interviews and may improve real time monitoring of suicides. We recommend a human-AI collaborative model, whereby deductive coding by the LLM is complemented by human inductive coding and further interpretation. Key messages • Integrating an LLM with qualitative research procedures is feasible in a collaborative model. • Integrating an LLM with qualitative research procedures allows near real-time monitoring based on qualitative data, which extends to other public health fields.

Evaluating large language models in analysing classroom dialogue

This study explores the use of Large Language Models (LLMs), specifically GPT-4, in analysing classroom dialogue—a key task for teaching diagnosis and quality improvement. Traditional qualitative methods are both knowledge- and labour-intensive. This research investigates the potential of LLMs to streamline and enhance this process. Using datasets from middle school mathematics and Chinese classes, classroom dialogues were manually coded by experts and then analysed with a customised GPT-4 model. The study compares manual annotations with GPT-4 outputs to evaluate efficacy. Metrics include time efficiency, inter-coder agreement, and reliability between human coders and GPT-4. Results show significant time savings and high coding consistency between the model and human coders, with minor discrepancies. These findings highlight the strong potential of LLMs in teaching evaluation and facilitation.

Abstract B127: Providers and survivors’ perspectives on affordability challenges for gastrointestinal cancer Black and White patients living in socioeconomically disadvantaged US states

Abstract Background: Affordability challenges for gastrointestinal (GI) cancer care may be numerous and complex in socioeconomically disadvantages states of the southern US, like Alabama (AL) and Mississippi (MS). Given that a large share of the African American US population lives in the southern US, these challenges may be contributing to observed racial disparities in GI cancer. Methods: Semi-structured interviews were conducted with purposefully selected AL and MS GI cancer providers, other staff, and survivors. Interview guides were developed based on access to care conceptual frameworks to identify barriers and facilitators to surgical and overall cancer care along 5 domains: Availability, Accessibility, Affordability, Acceptability, and Accommodation. Transcripts were analyzed by 3 coauthors using an inductive thematic and content analysis qualitative approach with NVivo 12 Plus software. Intercoder agreement was reached at 91%. This analysis focuses on the Affordability domain. Results: Providers (N=27) were 11 surgeons, 4 medical and radiation oncologists, 1 primary care provider, 6 nurses, and 5 patient navigators, 22% practiced in rural areas, 78% in urban areas. Survivors (N=36) were 44-87 years of age, 42% female, 70% white, 30% African-American, 53% colorectal, 19% pancreatic, 28% esophageal cancer. Providers and survivors discussed affordability challenges at the patient and system level. Patient-level challenges were: 1) High cost burden besides payments to providers (e.g., transportation, ostomy bags, nutritional supplements, non-antineoplastic drugs, feeding tubes, deductibles), and 2) Spending priorities outside of health care (e.g., patients prioritizing what to spend limited incomes on, which bills to pay). System-level affordability barriers included: 1) Care impacted by insurance processes (e.g., delays due to preauthorization or hospital out-of-network, surgery scheduled around deductible), and 2) Logistical burdens to access resources to help patients afford care (e.g., applying for grants). Two additional themes emerged from the providers’ interviews only: 1) Added costs due to scarcity of quality medical services (e.g., cost of repeating imaging procedures that were of poor quality), and 2) Poverty making it unfeasible to deliver guideline-concordant care (e.g., providers unable to provide recommended radiation treatment if patients need to travel long distances and have no money for gas). One more theme emerged from the survivors’ interviews only related to the burden of a confusing billing system (e.g., repeat billing for the same doctor). Conclusions: Many affordability-related challenges exist for the care of GI patients from disadvantaged states of the Deep South, and impact survivors and providers’ decision making about treatment and other needs. Future research should quantify the extent of these challenges and determine how they contribute to observed racial disparities in GI cancer surgery and other treatment. Citation Format: Maria Pisu, Nataliya V. Ivankova, Jessica Morgan, Yu-Mei M. Schoenberger, Nathan C. English, Burkely P. Smith, Ivan I. Herbey, Courtney Williams, Bayley A. Jones, Wendelyn M. Oslock, Daniel I. Chu. Providers and survivors’ perspectives on affordability challenges for gastrointestinal cancer Black and White patients living in socioeconomically disadvantaged US states [abstract]. In: Proceedings of the 17th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2024 Sep 21-24; Los Angeles, CA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2024;33(9 Suppl):Abstract nr B127.

Autoimmune rheumatic transitional care model development

AimTo develop a transitional care model for autoimmune rheumatic disease patients based on the needs analysis.MethodMixed Method, Explanatory sequential design (QUAN-qual) has been conducted. Quantitative data were collected through medical record and structured interviews. Qualitative study has been done through Focused Group Discussion (FGD), based on problems met in previous quantitative study. We have done the coding processed, followed by determining categories and themes to reach the intercoder agreement with peer-debriefing. Analysis of the final results of research was assisted by the external auditor to form a model of care.ResultThe quantitative data collection from 27 patients showed that the transition age was 18–19 year-old, age of onset 4–17 year-old, 23 patients (85, 2%) with SLE, 4 patients (14.8%) with JIA. Two patients (7.4%) had different diagnosis from the pediatric clinic, 1 patient (3.7%) had no diagnosis from previous clinic. Drug switching during transition occurred in 14 patients (51.9%) and 3 patients (11.1%) has no known medication history. Data regarding disease activity at initial diagnosis were not available in 26 patients (96.3%). The combined FGD analysis found several key words related to “the need of change” in RSCM autoimmune rheumatic transitional care.ConclusionA development of transitional care model for autoimmune rheumatic disease consist of documents about service algorithm, transfer documents, systematic work protocols with education check list has been done.

AI Application (ChatGPT) and Saudi Arabian Primary School Students’ Autonomy in Online Classes: Exploring Students and Teachers’ Perceptions

In education, the integration of artificial intelligence (AI) has presented opportunities to transform the dynamics of online learning. This study investigated the impact of an AI-powered application, namely ChatGPT, on the autonomy of Saudi Arabian primary students participating in online classes. It also explored how the implementation of Chat GPT influenced Saudi Arabian primary students’ autonomy. In this mixed-methods study, a quasi-experimental design assessed the impact of ChatGPT on learner autonomy among 250 Saudi Arabian primary students from six primary schools in Riyadh, Saudi Arabia. The quantitative analysis employed descriptive statistics and t-tests, while the qualitative data underwent interpretative phenomenological analysis. To ensure coding reliability, 20% of the codes were independently reviewed by an external coder, with a 94% inter-coder agreement coefficient reached through consensus. Findings revealed that ChatGPT significantly affected the participants’ perceptions of autonomy and its different dimensions. Qualitative data showed that AI-powered applications contributed to the students’ autonomy in 10 different ways. Participants also mentioned that AI-powered apps might have some negative consequences. This study has theoretical implications for redefining learner autonomy in the digital age and calls for the exploration of many facets of autonomy. Practical applications from this study include strategic integration of AI into online education, data security, and the need for orientation programs.

Inter-Coder Agreement in Qualitative Coding: Considerations for its Use

<i>The historically quantitative-dominated field of health sciences has increasingly embraced qualitative methods. However, calls for quantitative measures of rigor, such as Inter-coder Agreement (ICA), remain. The aim of this manuscript is to demystify ICA and provide practical guidance. I begin by describing considerations while planning for ICA, including differences between various ICA tests (i.e., percent agreement, Holsti Method, Cohen’s kappa, Krippendorf’s alpha, and Gwet’s AC1 and AC2), setting the threshold of acceptability for your chosen test, deciding whether to use qualitative data analysis software, choosing the number of coders, selecting what data will be coded by more than one coder, developing a deductive codebook, creating a process for resolving coding disagreements, and establishing an audit trail for codebook changes. Next, I provide step-by-step guidance on an iterative process used for enacting ICA. Finally, I discuss the importance of reporting, emphasizing clarity, conciseness, completeness, and accuracy.</i>

ChatGPT for Automated Qualitative Research: Content Analysis.

Data analysis approaches such as qualitative content analysis are notoriously time and labor intensive because of the time to detect, assess, and code a large amount of data. Tools such as ChatGPT may have tremendous potential in automating at least some of the analysis. The aim of this study was to explore the utility of ChatGPT in conducting qualitative content analysis through the analysis of forum posts from people sharing their experiences on reducing their sugar consumption. Inductive and deductive content analysis were performed on 537 forum posts to detect mechanisms of behavior change. Thorough prompt engineering provided appropriate instructions for ChatGPT to execute data analysis tasks. Data identification involved extracting change mechanisms from a subset of forum posts. The precision of the extracted data was assessed through comparison with human coding. On the basis of the identified change mechanisms, coding schemes were developed with ChatGPT using data-driven (inductive) and theory-driven (deductive) content analysis approaches. The deductive approach was informed by the Theoretical Domains Framework using both an unconstrained coding scheme and a structured coding matrix. In total, 10 coding schemes were created from a subset of data and then applied to the full data set in 10 new conversations, resulting in 100 conversations each for inductive and unconstrained deductive analysis. A total of 10 further conversations coded the full data set into the structured coding matrix. Intercoder agreement was evaluated across and within coding schemes. ChatGPT output was also evaluated by the researchers to assess whether it reflected prompt instructions. The precision of detecting change mechanisms in the data subset ranged from 66% to 88%. Overall κ scores for intercoder agreement ranged from 0.72 to 0.82 across inductive coding schemes and from 0.58 to 0.73 across unconstrained coding schemes and structured coding matrix. Coding into the best-performing coding scheme resulted in category-specific κ scores ranging from 0.67 to 0.95 for the inductive approach and from 0.13 to 0.87 for the deductive approaches. ChatGPT largely followed prompt instructions in producing a description of each coding scheme, although the wording for the inductively developed coding schemes was lengthier than specified. ChatGPT appears fairly reliable in assisting with qualitative analysis. ChatGPT performed better in developing an inductive coding scheme that emerged from the data than adapting an existing framework into an unconstrained coding scheme or coding directly into a structured matrix. The potential for ChatGPT to act as a second coder also appears promising, with almost perfect agreement in at least 1 coding scheme. The findings suggest that ChatGPT could prove useful as a tool to assist in each phase of qualitative content analysis, but multiple iterations are required to determine the reliability of each stage of analysis.

Provider perspectives on screening and treatment for opioid use disorder and mental health in HIV care: A qualitative study.

HIV, opioid use disorder (OUD), and mental health challenges share multiple syndemic risk factors. Each can be effectively treated with routine outpatient appointments, medication management, and psychosocial support, leading implementers to consider integrated screening and treatment for OUD and mental health in HIV care. Provider perspectives are crucial to understanding barriers and strategies for treatment integration. We conducted in-depth qualitative interviews with 21 HIV treatment providers and social services providers (12 individual interviews and 1 group interview with 9 participants) to understand the current landscape, goals, and priorities for integrated OUD, mental health, and HIV care. Providers were purposively recruited from known clinics in Mecklenburg County, North Carolina, U.S.A. Data were analyzed using applied thematic analysis in the NVivo 12 software program and evaluated for inter-coder agreement. Participants viewed substance use and mental health challenges as prominent barriers to engagement in HIV care. However, few organizations have integrated structured screening for substance use and mental health into their standard of care. Even fewer screen for opioid use. Although medication assisted treatment (MAT) is effective for mitigating OUD, providers struggle to connect patients with MAT due to limited referral options, social barriers such as housing and food insecurity, overburdened staff, stigma, and lack of provider training. Providers believed there would be clear benefit to integrating OUD and mental health treatment in HIV care but lacked resources for implementation. Integration of screening and treatment for substance use and mental health in HIV care could mitigate many current barriers to treatment for all three conditions. Efforts are needed to train HIV providers to provide MAT, expand resources, and implement best practices.

Providing an innovative model for social customer relationship management: Meta synthesis approach

Social Customer Relationship Management (SCRM) has revolutionized the interaction between companies and customers, creating new opportunities and challenges for both parties. This study aims to present an innovative model for identifying the dimensions and components of SCRM. The data were collected based on a qualitative research design utilizing the meta-synthesis method, following the seven steps outlined by Sandelowski and Barroso (2007). Considering the relevance to the research topic and the quality of sources, 31 studies were selected for the final evaluation. To assess the reliability of the findings, the kappa coefficient for inter-coder agreement was used, resulting in a kappa value of 0.79. The results, analyzed using MAXQDA20 software, 2 dimensions, 6 categories, and 16 concepts identified. The company-focused dimension included 3 categories: factors influencing adoption, inhibitors, and outcomes of implementation. Similarly, the customer-focused dimension identified 3 categories: factors influencing adoption, inhibitors, and outcomes of implementation. Ultimately, a comprehensive model for SCRM was developed. The results of this research can be utilized by academic references as well as companies and consumers.

Providers’ and survivors’ perspectives on the availability and accessibility of surgery in gastrointestinal cancer care

BackgroundSurgery is essential for gastrointestinal (GI) cancer treatment. Many patients lack access to surgical care that optimizes outcomes. Scarce availability and/or low accessibility of appropriate resources may be the reason for this, especially in economically disadvantaged areas. This study aimed to investigate providers’ and survivors’ perspectives on barriers and facilitators to the availability and accessibility of surgical care. MethodsSemistructured interviews informed by surgical disparities and access-to-care conceptual frameworks with purposively selected GI cancer providers and survivors in Alabama and Mississippi were conducted. Survivors were within 3 years of diagnosis of stage I to III esophageal, pancreatic, or colorectal cancer. Transcripts were analyzed using inductive thematic and content analysis techniques. Intercoder agreement was reached at 90 %. ResultsThe 27 providers included surgeons (n = 11), medical oncologists (n = 2), radiation oncologists (n = 2), a primary care physician (n = 1), nurses (n = 8), and patient navigators (n = 3). This study included 36 survivors with ages ranging from 44 to 87 years. Of the 36 survivors, 21 (58.3 %) were male, and 11 (30.6 %) identified as Black. Responses were grouped into 3 broad categories: (i) transportation/geographic location, (ii) specialized care/testing, and (iii) patient-/provider-related factors. The barriers included lack and cost of transportation, reluctance to travel because of uneasiness with urban centers, low availability of specialized care, overburdened referral centers, provider-related referral biases, and low health literacy. Facilitators included availability of charitable aid, centralizing multidisciplinary care, and efficient appointment scheduling. ConclusionIn the Deep South, barriers and facilitators to the availability and accessibility of GI surgical cancer care were identified at the health system, provider, and patient levels, especially for rural residents. Our data suggest targets for improving the use of surgery in GI cancer care.

Early career researchers in health policy and systems research: insights from freelancers in a non-profit organization in the Philippines

BackgroundThe freelance economy has seen rapid growth worldwide in recent years and the Philippines is not an exception. Freelance workers are becoming increasingly common in healthcare and research. Early career researchers carry out most of scientific research and can play a critical role in advancing public health by bringing new perspectives and diversity to the field. Existing literature has mostly focused on the experiences of early career researchers in an institutional academic setting. This study aimed to understand the experiences of freelance early career researchers in the health policy and systems space in the Philippines.MethodsThis qualitative study collected data from 18 to 22 March 2022 through virtual interview and focus group discussions. Themes and codes were created based on the topic guide developed. New themes and codes were generated as they emerged. Two researchers coded the data using both a priori and emergent codes. Any coding conflicts were resolved through discussions until intercoder agreement was reached. Interpretation and conclusions from the data were developed by 2 researchers with consideration for its context and relationship between themes.ResultsFifteen current and former freelance researchers participated in the study. Most are female, under 35 years old, and with an undergraduate degree as the highest educational attainment. The findings highlight insights and challenges faced by early career researchers in aspects of: (1) work arrangement, (2) tasks, (3) expectations from senior researchers, (4) development in the health policy and systems field, (5) relationship with peers, and (6) motivations for continuing to work as a freelance health policy and systems researcher.ConclusionThis study reveals the challenges freelance early career researchers face, highlighting the need for enhanced support and recognition amidst rapidly evolving workforce demands and complex health dilemmas. Recommendations include structured mentorship, professional development, innovative funding models, and the establishment of a supportive network. Advocacy for policies ensuring freelancer inclusion in the economy and policy-making is crucial. Future research should investigate their experiences further, including their roles, transitions, and the impacts of funding trends, to foster their development and integration into public health research and policy.

Occupational therapy students' perceptions of their experience in a role-emerging Level II fieldwork within higher education student services

BackgroundRole-emerging settings – those where occupational therapy (OT) services have not traditionally been provided – are common sites for practice placements of entry-level occupational therapy students. A growing body of literature has attempted to determine the value and drawbacks of such practice placements on the professional preparedness of OT students with mixed findings. Benefits have been identified, including increased cultural understanding, advocacy, creativity, initiative, and problem-solving skills. However, OT students have been reported to perceive such placement as limiting their professional growth and preparedness to practice compared to traditional placements.MethodsA phenomenological study was conducted seeking the perceptions of OT students (n = 14) about their clinical placement at a role-emerging site. Recorded semi-structured interviews were conducted by trained interviewers within two weeks of the end of clinical placement. The recordings were transcribed verbatim and then coded using an iterative multi-coder inductive approach. Inter-coder agreement, reflectivity, and audit trail were maintained.ResultsThree themes emerged from the analysis: (1) integrating independence and support, (2) becoming occupational therapists, and (3) filling a gap. These themes reflect students’ positive perceptions of their role-emerging clinical placement. They felt that this placement allowed them to develop self-confidence and professional identity as occupational therapists and learn new skills while simultaneously filling a gap in services for clients. Most importantly, they felt that this placement prepared them for their future OT practice.ConclusionThis finding and their resounding support of the experience suggest that OT students can perceive role-emerging placement as a solid foundation for clinical practice. Factors, included in this placement, that may have contributed to their experience include the level of support provided, time available for learning including space to make mistakes, and freedom from productivity and payor requirements.

105 Gender-Diverse Inclusion: The Language of Sex and Gender in PrEP Clinical Trials

OBJECTIVES/GOALS: To construct an assessment scale capable of evaluating a trial’s gender literacy or the extent to which biologically assigned “sex” is understood as separate from culturally defined and personally embodied “gender”. This scale in tandem with a policy brief will outline recommendations for inclusive medical nomenclature in the clinical space. METHODS/STUDY POPULATION: Using clinicaltrials.gov, inclusion/exclusion criteria was recorded for PrEP interventional trials (i.e., Truvada, Descovy). To evaluate these trials, an assessment scale for “gender literacy” is necessary. This scale relies on the fact that sex and gender are distinct elements to one’s identity and ought to be reported as such. As a form of content analysis, where literary information (eligibility criteria) is evaluated based on set rubric, this scale will require validation through inter-coder agreement. Evaluated in a group of 5 college-age students, this scale was used on selected PrEP clinical trials to verify if there was high agreement in the scores given. After validation, the dataset from clinicaltrials.gov underwent evaluation using the proposed assessment scale for gender literacy. RESULTS/ANTICIPATED RESULTS: The student coders had a Kalpha of 0.4 in the first round of grading. After retraining, their Kalpha increased to 0.68. The grading involved a subjective language rating (LIR), evaluating the usage of inclusive language, and a numerical score (GR) for the demographics of inclusion in a trial. After this inter-coder agreement validation, 216 active PrEP clinical trials (as of March 2023) were downloaded from clinicaltrials.gov. Grading of these trials showed that cisgender males represented 40% of participants, while 28% represented both transgender men and women, and less than 1% represented non-binary individuals. Moreover, more than half of the trials (52%) exhibited cisgender-oriented language or made no reference to gender identity. DISCUSSION/SIGNIFICANCE: It is a scientific imperative for clinical trials to have representative participant bases in order to derive data that is generalizable to afflicted populations. Especially for PrEP clinical trials, where gender-diverse individuals need visibility, trial design must be carefully crafted so as not to exclude through dated or exclusionary language.

Are multiple coders needed for cause of death assignment: results from telephonic verbal autopsy interviews conducted in 2021 in South Africa.

Verbal autopsy (VA) methods have emerged to estimate causes of death in populations lacking robust civil registration and vital statistics (CRVS) systems. Despite World Health Organization endorsement of routine VA use, cost and efficiency concerns persist. Telephonic verbal autopsies (teleVAs) can reduce cost. Physician coding offers a valuable approach, but the expertise required makes it resource-intensive, often involving multiple coders for consensus. To assess inter-coder agreement for cause of death (CoD) in South African teleVAs using Kappa statistics, evaluating if agreement surpasses a 0.8 cut-off (very high) potentially allowing single coders. A cross-sectional study employed telephonic VA interviews on non-facility deaths in Cape Town (December 2020-September 2021). Trained fieldworkers administered a standard VA questionnaire. Each case's VA responses were reviewed independently by two physicians, medically certifying the CoD. A panel was used to solve disagreements. Cohen's kappa-statistic (k-statistic) tested agreement levels. Decedents were aged between 18 and 98 years. In total, 228 teleVAs (16.6% response rate) were conducted. Physician coding agreement was good overall (k-statistic: 0.63). Diabetes mellitus (47%) and other non-communicable disease (42%) had initial agreement between physician coders in less than 50% of cases in comparison to consensus totals. COVID-19 (89%) and acute cardiac disease (83%) showed initial agreement in more than 80% of cases compared to consensus totals. A chi-square test revealed a significant difference in the number of causes listed on death notification forms for cases with and without agreement in Part 1 (χ2 = 14.71, p < 0.01), but not in Part 2 (χ2 = 4.97, p = 0.17). CoD agreement might not be high enough to infer that single coders can be used instead of multiple coders. Challenges with co-morbidities and specific CoDs with multiple sequelae highlight the need for further research and refinement of VA methodologies for reliable CoD determination in routine practice.

Understanding Teachers’ Ethnic-Racial Socialization Practices with Students in Schools: A Qualitative Inquiry

Ethnic-racial socialization (ERS) is essential for youth of color to navigate the racialized world. There is a need to understand teachers’ practices as an extension of family-based ERS. This study explores teachers’ ERS engagement with African American and Latine adolescents attending two large, diverse high schools. Two 90-minute focus groups were conducted ( n = 15), utilizing thematic analysis and inter-coder agreement. Teachers reported nuanced ERS messages and methods. Facilitators and barriers ranged from macro (i.e., institutional), meso (i.e., community), and micro-level (i.e., personal). Findings are understood within congruency or distinction from parental ERS with implications for teacher practices within urban schools.

Designing a Model for the Spiritual Elevation of Students with an Emphasis on Outcomes

Objective: The application of spirituality in today's fast-paced and concern-filled life; attention to the transcendent aspect, inner truth, meaning-making, and deep thinking that, by shaping in the minds of students, leads to the development of security, peace, the spread of joy and contagious happiness among the community, and its spiritual elevation. The present study aimed to design a model for the spiritual elevation of students with an emphasis on outcomes. Methods and Materials: The method of the current study was qualitative and utilized an interpretative content analysis approach. For this purpose, findings from the review of literature and theoretical foundations were aligned with the results from the analysis of semi-structured interviews conducted purposively and optimally with members of the educational science group at universities and higher education centers in Lorestan province in 2023. The research tool was semi-structured interviews, for which validity was assessed using a formal method, and reliability was measured using an inter-coder agreement coefficient, which resulted in 87%. Data analysis was also conducted using a coding method. Findings: After analyzing the data, eventually, eight indicators based on the outcomes formed the model of students' spiritual elevation, which included insight, faith, social responsibility, spiritual self-awareness, spiritual aesthetics, purposefulness, meaning-centeredness, and ethics-orientedness. Conclusion: The use of the findings of this article can practically play a role in fostering the spirit of spirituality among students in a real and non-abstract manner, and its extension to society can be beneficial.

The use of social network analysis in educational sciences studies

Since social networks analysis in education offers valuable insights into social structures and social dynamics that shapes individuals behaviors and information storage and transmission, it has become a hot topic in educational science studies. The aim of this study is to examine the educational sciences studies conducted at higher education level in which social network analysis is used. The studies were analyzed based on journals, years, author countries, number of citations, models, theories, and concepts, research methods and target audience. Content analysis method was used in the study. The reliability of inter-coder agreement was calculated as .88. The findings were categorized under certain themes according to the research questions. According to the results, Internet and Higher Education (n=6) and Computers and Education (n=5) were the journals with the most publications, while 2019 was the year that the most studies (n=12) were conducted. The studies were mostly conducted by authors in the USA. "Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking" was the most cited article. When the underlying models, theories and concepts in the studies were analyzed, six themes emerged: social paradigm, learning environments/tools, learning approaches/methods, feedback/assessment, informal approaches to teaching and individual characteristics. The most frequently used method was quantitative research, and the target group was undergraduate students. The target group size was mostly between 30-60, and convenience sampling was primarily employed for the target group selection. According to the findings and results of the study, suggestions for the use of social network analysis in the field of educational sciences were presented.

Harmonizing DevOps taxonomies — A grounded theory study

Context:DevOps responds to the growing need of companies to streamline the software development process and thus has experienced widespread adoption in the past few years. However, the successful adoption of DevOps requires companies to address important cultural and organizational changes. Nevertheless, it is crucial to recognize that various DevOps taxonomies exist, both from academic and practitioner perspectives, which may lead to misleading or failed adoption of DevOps. Objective:This paper presents empirical research on the structure of DevOps teams in software-producing organizations. The goal is to better understand the organizational structure and characteristics of teams adopting DevOps by harmonizing the existing knowledge. Methods:To achieve this, we employed a grounded theory approach with collaborative coding, involving two research groups. Inter-Coder Agreement (ICA) was utilized to guide the discussion rounds. We conducted a comprehensive analysis of existing studies on DevOps teams and taxonomies to gain a deeper understanding of the subject. Results:From the analysis, we built a substantive and analytic theory of DevOps taxonomies. The theory is substantive in that the scope of validity refers to the ten secondary studies processed and analytic in that it analyzes “what is” rather than explaining causality or attempting predictive generalizations. A public repository with all the data related to the products resulting from the analysis and generation of the theory is available. Conclusions:We built a theory on DevOps taxonomies and tested whether it harmonizes the existing taxonomies, i.e., whether our theory can instantiate the others. This is the first step to define which taxonomies are best suited to approach DevOps culture and practices according to the companies’ objectives and capabilities.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Patient (pt) Experiences of Receiving Idecabtagene Vicleucel (Ide-Cel, bb2121) Versus Standard (Std) Regimens for the Treatment (Tx) of Relapsed/Refractory Multiple Myeloma (RRMM) in the Randomized, Controlled KarMMa-3 Clinical Trial: Analysis of Longitudinal Qualitative Interviews

Introduction: Ide-cel is a first-in-class B-cell maturation antigen (BCMA)-directed chimeric antigen receptor T cell therapy approved by the United States (US) Food and Drug Administration; the European Medicines Agency; and Japan's Ministry of Health, Labour, and Welfare for treating adult pts with RRMM who were triple-class exposed (TCE) to an immunomodulatory (IMiD) agent, proteasome inhibitor (PI), and anti-CD38 monoclonal antibody and who experienced disease progression/relapse after last therapy. The phase 3 multicenter, open-label, randomized controlled KarMMa-3 trial (NCT03651128) enrolled pts with RRMM who were TCE (including an IMiD agent, PI, and daratumumab [DARA]) after 2-4 prior lines of therapy. Pts were randomized 2:1 to receive either ide-cel or 1 of 5 std regimens (DARA, pomalidomide [POM], dexamethasone [DEX]; DARA, bortezomib, DEX; ixazomib, lenalidomide, DEX; carfilzomib and DEX; elotuzumab, POM, DEX). At median follow-up (18.6 mo), median progression-free survival (13.3 vs 4.4 mo) and response rate (71% vs 42%) were significantly higher in the ide-cel versus std regimen arm (Rodriguez-Otero P, et al. N Engl J Med 2023;388:1002-1014). In addition to the standard pt-reported outcomes instruments, trial-embedded interviews provide further insights into the patient Tx experience (Delforge M, et al. J Clin Oncol 2023;41[suppl 16]. Abstract 8032). This study assessed Tx experience, changes in health-related quality of life, and advantages and disadvantages of assigned Tx for pts enrolled in the KarMMa-3 trial. Methods: To obtain pts' experiences and perspectives in their own words while on the assigned Tx arms during the study, qualitative interviews were embedded into the KarMMa-3 trial. Pts were randomized (1:4) to participate in a series of optional qualitative interviews conducted at baseline (BL), and 3 and 6 mo post Tx. Interviews were audio-recorded and transcribed. Non-English interviews were transcribed directly into English. Transcripts were analyzed in MAXQDA qualitative analysis software; inter-coder agreement exercises were completed to ensure reliability. Analytic approaches included thematic analysis and longitudinal analysis across aspects of health and well-being. Results: In total, 64 pts (n = 43 ide-cel, n = 21 std regimen) across 9 countries in Europe and in the USA participated in ≥ 1 interview. Across both arms, 60, 48, and 38 pts completed interviews at BL, and 3 and 6 mo post Tx, respectively. The mean (median) age of interviewed pts at time of consent in the trial was 60.5 (60) years, and 33% were female. When describing their overall well-being during the previous 4 weeks at 3 mo, 53% of ide-cel pts described their well-being as mostly improved versus 19% in the std regimen arm. In the ide-cel arm, 59% and 56% of pts reported improvements in physical health versus 19% and 10% of pts in the std arm at 3 and 6 mo from BL, respectively; 25% and 28% of pts in the ide-cel arm reported worsening physical health at 3 and 6 mo from BL versus 63% and 80% of pts in the std arm (Table 1). In the ide-cel arm, 47% and 48% of ide-cel pts described improvements in physical functioning at 3 and 6 mo from BL versus 19% and 20% of std regimen pts, respectively; 22% and 16% of pts in the ide-cel arm versus 50% and 60% of pts in the std arm reported worse physical functioning at 3 and 6 mo from BL. Regarding Tx advantages, ide-cel pts reported efficacy (n = 25, 58%), minimal side effects (n = 21, 49%), avoidance of other therapies (n= 16, 37%), and having a 1-time Tx (n = 12, 28%). Std regimen pts also described efficacy (n = 10, 48%) and minimal side effects (n = 8, 38%) as Tx advantages. Disadvantages discussed by pts treated with ide-cel included lack of efficacy (n = 10, 23%), side effects after infusion (n = 9, 21%), or none (n = 9, 21%). Greater proportions of std regimen pts described side effects (n = 9, 43%) and lack of efficacy (n = 7, 33%). Additional well-being results for both arms are presented in Table 1. Conclusions: This study provides unique insight into key pt perceptions of ide-cel Tx compared to std regimens. Overall, pts receiving ide-cel reported more positive changes 6 mo post Tx than pts receiving std regimens, specifically in physical health and functioning. Fewer pts treated with ide-cel described side effects than pts receiving std regimens. These data show that ide-cel as a 1-time infusion can alleviate the burden of managing complex medication regimens in TCE early-line relapse pts.

An mHealth Platform for People With HIV Receiving Care in Washington, District of Columbia: Qualitative Analysis of Stakeholder Feedback.

HIV viral suppression and retention in care continue to be challenging goals for people with HIV in Washington, District of Columbia (DC). The PositiveLinks mobile app is associated with increased retention in care and viral load suppression in nonurban settings. The app includes features such as daily medication reminders, mood and stress check-ins, an anonymized community board for peer-to-peer social support, secure messaging to care teams, and resources for general and clinic-specific information, among other features. PositiveLinks has not been tailored or tested for this distinct urban population of people with HIV. This study aimed to inform the tailoring of a mobile health app to the needs of people with HIV and their providers in Washington, DC. We conducted a 3-part formative study to guide the tailoring of PositiveLinks for patients in the DC Cohort, a longitudinal cohort of >12,000 people with HIV receiving care in Washington, DC. The study included in-depth interviews with providers (n=28) at study clinics, focus groups with people with HIV enrolled in the DC Cohort (n=32), and a focus group with members of the DC Regional Planning Commission on Health and HIV (COHAH; n=35). Qualitative analysis used a constant comparison iterative approach; thematic saturation and intercoder agreement were achieved. Emerging themes were identified and grouped to inform an adaptation of PositiveLinks tailored for patients and providers. Emerging themes for patients, clinic providers, and COHAH providers included population needs and concerns, facilitators and barriers to engagement in care and viral suppression, technology use, anticipated benefits, questions and concerns, and suggestions. DC Cohort clinic and COHAH provider interviews generated an additional theme: clinic processes. For patients, the most commonly discussed potential benefits included improved health knowledge and literacy (mentioned n=10 times), self-monitoring (n=7 times), and connection to peers (n=6 times). For providers, the most common anticipated benefits were improved communication with the clinic team (n=21), connection to peers (n=14), and facilitation of self-monitoring (n=11). Following data review, site principal investigators selected core PositiveLinks features, including daily medication adherence, mood and stress check-ins, resources, frequently asked questions, and the community board. Principal investigators wanted English and Spanish versions depending on the site. Two additional app features (messaging and documents) were selected as optional for each clinic site. Overall, 3 features were not deployed as not all participating clinics supported them. Patient and provider perspectives of PositiveLinks had some overlap, but some themes were unique to each group. Beta testing of the tailored app was conducted (August 2022). This formative work prepared the team for a cluster randomized controlled trial of PositiveLinks' efficacy. Randomization of clinics to PositiveLinks or usual care occurred in August 2022, and the randomized controlled trial launched in November 2022. RR2-10.2196/37748.