Intercoder Agreement Research Articles

Exploring Muslim Women's Reproductive Health Needs and Preferences in the Emergency Department.

Objective: We explored individual Muslim women's reproductive healthcare experiences, preferences, beliefs, and behaviors in the emergency department (ED) and in general. Methods: This was a qualitative study conducted at a community ED using semi-structured interviews with a piloted interview guide. We interviewed participants awaiting care in the ED with the following criteria: female gender; English or Arabic speaking; aged ≥18years; and self-identified as Muslim. We conducted interviews in both English and Arabic until thematic saturation was reached. Transcripts were coded using an iteratively developed codebook, maintaining intercoder agreement greater than 80%. We used an inductive thematic analysis to identify themes, and results were interpreted in the context of interview language and patient's age. Results: We interviewed 26 Muslim-identified female ED patients. We found that cultural representation and sensitivity among ED staff mitigated discrimination and promoted inclusion for Muslim ED patients. However, assumptions about Muslim identity also impacted the participants' healthcare. Most participants endorsed a preference for a female clinician for their reproductive healthcare in general, but not necessarily for other areas of medicine. Clinician cultural concordance was not always preferred for participants in the ED due to fears about the loss of confidentiality. Marital status impacted beliefs about reproductive and sexual health in the context of Muslim identity. Overall, family planning was acceptable and encouraged in this patient population. Conclusion: The themes elucidated in this study may guide clinicians in developing culturally sensitive practices when providing reproductive healthcare to the Muslim population.

Assessing ad-hoc adaptations’ alignment with therapeutic goals: a qualitative study of lay counselor-delivered family therapy in Eldoret, Kenya

BackgroundA key question in implementation science is how to balance adaptation and fidelity in translating interventions to new settings. There is growing consensus regarding the importance of planned adaptations to deliver interventions in contextually sensitive ways. However, less research has examined ad-hoc adaptations, or those that occur spontaneously in the course of intervention delivery. A key question is whether ad-hoc adaptations ultimately contribute to or detract from intervention goals. This study aimed to (a) identify ad-hoc adaptations made during delivery of a family therapy intervention and (b) assess whether they promoted or interrupted intervention goals.MethodsTuko Pamoja (Swahili: “We are Together”) is an evidence-informed family therapy intervention aiming to improve family dynamics and mental health in Kenya. Tuko Pamoja employs a task-shifting model, delivered by lay counselors who are afforded a degree of flexibility in presenting content and in practices they use in sessions. We used transcripts of therapy sessions with 14 families to examine ad-hoc adaptations used by counselors. We first identified and characterized ad-hoc adaptations through a team-based code development, coding, and code description process. Then, we evaluated to what extent ad-hoc adaptations promoted the principles and strategies of the intervention (“TP-promoting”), disrupted them (“TP-interrupting”), or neither (“TP-neutral”). To do this, we first established inter-coder agreement on application of these categories with verification by the intervention developer. Then, coders categorized ad-hoc adaptation text segments as TP-promoting, TP-interrupting, or TP-neutral.ResultsAd-hoc adaptations were frequent and included (in decreasing order): incorporation of religious content, exemplars/role models, community dynamics and resources, self-disclosure, and metaphors/proverbs. Ad-hoc adaptations were largely TP-promoting (49%) or neutral (39%), but practices were TP-interrupting 12% of the time. TP-interrupting practices most often occurred within religious content and exemplars/role models, which were also the most common practices overall.ConclusionExtra attention is needed during planned adaptation, training, and supervision to promote intervention-aligned use of common ad-hoc adaptation practices. Discussing them in trainings can provide guidance for lay providers on how best to incorporate ad-hoc adaptations during delivery. Future research should evaluate whether well-aligned ad-hoc adaptations improve therapeutic outcomes.Trial registrationPilot trial registered at clinicaltrials.gov (C0058)

Assessing the Implementation of University Management Strategies for Combating Examination Misconduct in Tanzanian Universities

This study assessed the implementation of university management strategies for combating examination misconduct in Tanzanian universities. It was guided by the Theory of Planned Behaviour developed by Ajzen in 1991. The study employed a convergent research design under mixed methods research approach. The target population was 27 universities, 51,489 students, 708 lecturers, 27 Chief examination officers and 27 Directors of quality assurance from which the sample size of 460 was selected through probability and non-probability sampling procedures. A questionnaire, document analysis guide and interview guide were used to collect data. Source triangulation and research experts ensured validity. Reliability was ensured through Cronbach alpha and intercoder agreement techniques. Quantitative data was analysed using descriptive and inferential statistics whereas qualitative data was analysed through thematic analysis. The study found that strategies such as penalties for examination offenders and checking for student’ illegal material were moderately implemented and there was a significant relationship between the levels of implementation of university management strategies for combating examination misconduct and the occurrence of various forms of examination misconduct in Tanzanian universities. The study concluded that university management strategies are not effectively implemented and there is a significant relationship between the levels of implementation of university management strategies and the occurrence of various forms of examination misconduct. The study recommends that universities in Tanzania in collaboration with Tanzania Commission for Universities should review and/or formulate academic integrity policies that will specify and uphold universities’ ethical principles and values as well as practices of appropriate examination conduct for handling policy violations to realize quality examinations

Examining the Prevalent Forms of Examination Misconduct in Tanzanian Universities

This study examined the prevalent forms of examination misconduct in Tanzanian universities. It was guided by the Theory of Planned Behaviour developed by Ajzen in 1991. The study employed convergent research design under mixed methods research approach. The target population consisted of 27 universities, which offer Bachelor of education degree in Tanzania mainland, 51,489 students, 708 lecturers, 27 Chief examination officers and 27 Directors of quality assurance from which the sample size of of 460 was selected through purposive and stratified random sampling techniques. Questionnaires, document analysis guide and interview guide were used to collect data. Source triangulation and research experts ensured validity. Reliability was ensured through Cronbach alpha and intercoder agreement techniques. Quantitative data was analysed using descriptive and inferential statistics whereas qualitative data was analysed through thematic analysis. The study found out that, the common forms of examination misconduct included the use of unauthorized materials, copying, collusion, examination leakage and impersonation. The study concluded that, there are diverse forms of examination misconduct in Tanzanian universities. The study recommends that, universities in Tanzania should provide moral education to students to sensitize them about the negative impact of embracing academic misconduct.

What Do We Need? Accommodations in the Workplace: People with Lupus Share Their Work Experiences

This article discusses disability inclusion in the workplace. It identifies barriers and makes recommendations to help meet employment needs of people with disabilities, utilizing lupus as an example. A convergent mixed method design study examined the workplace experiences and needs of individuals with lupus. Availability sampling yielded 1,776 participants from across the United States. The results revealed 69% of the respondents reported lupus symptoms had a negative impact on their employment and 48% of the participants reported needing help or accommodations in the workplace. An inductive thematic content analysis with an inter-coder agreement ranging from 71% to 73%, yielded 4 themes encompassing issues related to productivity, interpersonal relationship in the workplace, and adjustments in work responsibilities and work environment, respectively. The results suggest that training programs about the American with Disability Act may have a positive impact on organizational functioning. Employers have a responsibility to maintain a flexible work environment whenever possible to address variability in employees’ needs. Employers and employees should engage in ongoing discussions that seek to identify the accommodations that will help employees with disabilities meet their workplace responsibilities and to enhance their contribution to the organization. Managers and administrators are encouraged to create an environment that values a diverse workforce, including persons with disabilities.

A socioecological qualitative analysis of barriers to care in colorectal surgery

BackgroundAlthough specific social determinants of health have been associated with disparities in surgical outcomes, there exists a gap in knowledge regarding the mechanisms of these associations. Gaining perspectives from multiple socioecological levels can help elucidate these mechanisms. Our study aims to identify social determinants of health that act as barriers or facilitators to surgical care among colorectal surgery stakeholders. MethodsWe recruited participants representing 5 socioecological levels: patients (individual); caregivers/surgeons (interpersonal); and leaders in hospitals (organizational), communities (community), and government (policy). Patients participated in focus groups, and the remaining participants underwent individual interviews. Semistructured interview guides were used to explore barriers and facilitators to surgical care at each socioecological level. Transcripts were analyzed by 3 coders in an inductive thematic approach with content analyses. The intercoder agreement was 93%. ResultsSix patient focus groups (total n = 18) and 12 key stakeholder interviews were conducted. The mean age of patients was 54.7 years, 66% were Black, and 61% were female. The most common diseases were colorectal cancer (28%), inflammatory bowel disease (28%), and diverticulitis (22%). Key social determinants of health impacting surgical care emerged at each level: individual (clear communication, mental stress), interpersonal (provider communication and trust, COVID-related visitation restrictions), organizational (multiple forms of contact, quality educational materials, scheduling systems, discrimination), community (community and family support and transportation), and policy (charity care, patient advocacy organizations, insurance coverage). ConclusionKey social determinants of health-impacting care among colorectal surgery patients emerged at each socioecological level and may provide targets for interventions to reduce surgical disparities.

Reliability in software engineering qualitative research through Inter-Coder Agreement

The research on empirical software engineering that uses qualitative data analysis is increasing. However, most of them do not deepen into the validity of the findings, specifically in the reliability of coding in which these methodologies rely on. This paper aims to establish a novel theoretical framework that enables a methodological approach for conducting this validity analysis through Inter-Coder Agreement (ICA), based on the use of coefficients to measure the degree of agreement in collaborative coding. We systematically review several existing variants of Krippendorff’s α coefficients and provide a novel common mathematical framework to unify them. Finally, this paper illustrates the use of this theoretical framework in a large case study on DevOps culture. We expect that this work will help researchers who are committed to measuring consensus with quantitative techniques in collaborative coding, conducted as part of a qualitative research, to improve the rigor of their findings.

Developing Shared Ways of Seeing Data: The Perils and Possibilities of Achieving Intercoder Agreement

All research, e.g., qualitative or quantitative, is concerned with the extent to which analyses can adequately describe the phenomena it seeks to describe. In qualitative research, we use internal validity checks like intercoder agreement to measure the extent to which independent researchers observe the same phenomena in data. Researchers report indices of agreement to serve as evidence of consistency and dependability of interpretations, and we do so to make claims about the trustworthiness of our research accounts. However, few studies report the methods of how multiple analysts developed alignment in their interpretation of data, a process that undergirds accounts of consistency, dependability, and trustworthiness. In this article, we review the issues and options around achieving intercoder agreement. Drawing on our experience from a longitudinal, team-based research project that required rapid cycles of qualitative data analysis, we reflect on the challenges we had achieving high intercoder agreement (which refer to as the perils). It was through these challenges that we developed a method that helps to foster shared ways of seeing data, and thus alignment in our interpretations of phenomena in data. In this article, we present this method as a tool for dyadic and team-based qualitative data analysis to facilitate reliable and consistent high-inference interpretations of data with multiple analysts.

Identifying the Antecedents and Consequences of Health Leadership in the Hospitals of Lorestan University of Medical Sciences, Health and Medical Services

Objective: The purpose of this article was to investigate the antecedents and consequences of the conceptualized model of health leadership for the promotion of employee health in the hospitals of Lorestan University of Medical Sciences, Health and Medical Services. Methods: The research method employed was qualitative, using thematic analysis, and was applied in nature. The participants in the study included some experts in the field of management in the health sector, who were selected based on purposive sampling. Fourteen individuals were chosen based on specific criteria (management service history in the health sector, willingness to be interviewed, general management experiences, publications in the health and management fields). The sample size was also determined based on the principle of theoretical saturation. The research tool was semi-structured interviews, which were developed through several rounds of feedback with experts. Its validity was confirmed through expert opinions, and reliability was assessed through inter-coder agreement coefficient. Data were analyzed using coding method. Findings: The results indicated that for identifying the antecedents and consequences of health leadership in the hospitals of Lorestan University of Medical Sciences, there are four overarching themes (health leadership, health leadership mechanisms, antecedents of health leadership, and consequences of health leadership) and 22 organizing themes. Conclusion: The findings of the research provide an opportunity to respond to well-being and as a part of identifying significant prominent themes in human capital, particularly aiding and supporting management in reducing turnover and relocation for the health of employees.

Designing A Paradigmatic Model of Barriers to Innovation Management in Ahvaz Primary Schools

Objective: The purpose of the present study was to design a paradigmatic model of innovation management barriers in primary schools of Ahvaz city. Method: The research method was qualitative and, in terms of purpose, applied. The statistical population included experts in schools and universities, selected through purposive and snowball sampling. The research tool included semi-structured interviews, whose validity was established through face validity, and reliability was determined using the inter-coder agreement coefficient, resulting in 0.81. Data were analyzed using grounded theory method. Results: The findings revealed that causal conditions included understanding innovation characteristics, feeling the necessity for innovation, identifying individual differences among employees, managing information and communication technology, organizational culture, risk-taking in school employees, organizational structure, material resources, and transformative leadership; contextual conditions included individual characteristics of employees, employee experiences, individual characteristics of the manager, management style, organizational rules and regulations, organizational climate, and having an innovation model. Moreover, intervening conditions included psychological barriers, intolerance of ambiguity and failure, educational barriers, research barriers, performance evaluation barriers, structural barriers, human resource barriers, managerial barriers, and cultural barriers within the organization. Finally, strategic conditions included internal and external networking, division of labor, strategic management, committed and specialized human resources, politicization, having an innovative spirit among employees, the existence of a suggestion acceptance system in schools, administrative hierarchy, delegation of authority by the manager to subordinates, employee support, strategic planning, and the outcomes included increased mental health of employees and students, a creative school environment, a reward system for innovative and creative employees, a research-oriented culture, a learning organization, and improved educational quality. Conclusion: Based on the results, it can be said that the barriers to innovation management in primary schools of Ahvaz city are various factors, and the sub-system of education and management should prioritize and address these barriers.

Comparison of Undergraduate Social Studies Instruction Program Elective Course Content and Social Studies Curriculum (2018) Objectives

The 2018 Undergraduate Social Studies Instruction Program included 13 elective field courses. The pre-service teachers should select six elective courses to graduate. The present study aims to compare the content of the above-mentioned elective field courses and the 4th, 5th, 6th and 7th grade social studies curricula (2018) objectives. In the present qualitative study, the data were collected with the document analysis technique. The qualitative study data were analyzed with the Miles & Huberman (1994) reliability formula and the inter-coder agreement was determined as 91.4%. It was determined that 24 objectives in the Turkish Judicial System course, 21 objectives in the Turkish Economy course, 15 objectives in the Globalization and Society course, 10 objectives in the Environmental Education course, 9 objectives in the Historical Evidence in Social Studies Instruction, Local and Oral History, and Information Technologies in Social Studies courses, 8 objectives in the Maps and Applications course, 2 objectives in Current Global Issues, Regional Children’s Games in Social Studies Instruction, and Material Design in Social Studies Instruction courses were associated with the social studies curriculum objectives. The objectives specified in Evaluation of In-Classroom Learning, Social Studies Textbook Review, and Drama in Social Studies Instruction courses were not associated with curricular objectives.

AFRICAN-AMERICAN DEMENTIA CAREGIVER STRATEGIES INVENTORY (DCSI-A): INITIAL PSYCHOMETRIC EVALUATION

Abstract Recent studies have found promising evidence of the benefits of cognitive-behavioral interventions (CBIs) in improving depression and health status of African-American dementia caregivers (CGs). However, limited information exists about the specific CB strategies used in these investigations and which CB strategies were most effective in reducing caregiver distress and distress-related health problems. To address these shortcomings, Glueckauf and colleagues developed the African-American Dementia Caregiver Strategies Inventory (DCSI-A) using data from two CBI interventions (Nf129). The primary objectives of the current study were: (a) to identify the most frequently used categories of CBI strategies deployed by African-American CGs and (b) to conduct an initial evaluation of the psychometric properties of the DCSI-A. Descriptive statistics were calculated to assess strategy category frequency and percent of total responses. The two most frequently used strategies were: (1) CG Health-Related Intervention Strategies with 19 endorsements, 16.10% of total strategies and (2) Strategies for CG Social, Recreational, Personal Enhancement and Other Community Activities with 18 endorsements, 15.25% of total responses. These findings suggested that health-related and social, recreational, and personal enhancement strategies were the most commonly used by African-American CBI participants. Intercoder agreement was assessed across two one-month coding intervals, each of which contained 5 coding sessions. Mean percent of agreement and Kappa were .94 and .93, respectively. Both reliability measures fell within the highly acceptable range. Coder drift was minimal across the two time intervals. Future research will evaluate the psychometric properties of the DCSI-A with a larger sample of African-American CGs undergoing CBI.

Social Image of Old Age, Gendered Ageism and Inclusive Places: Older People in the Media.

Ageism promotes the exclusion of older people from society by generating a negative image that they also internalize. The aim of this article is to investigate older people's social self-image, through statements broadcast on a national Spanish radio program aimed at this group. A qualitative analysis was conducted for a random sample from the sound archive for the Radio Nacional de España program Juntos Paso a Paso (Together, Step by Step) (2008-2021), using codes based on the pillars and determinants of active ageing and the three dimensions of ageism. Intercoder agreement was calculated. There were significant findings regarding ageism, gendered ageism and ageing in place, with differences according to size of municipal area. The program in question can be considered a viable secondary source for the research aim. Ageism is most commonly manifested through implicit opinions and invisibilization in family and social contexts. Care activities play a notably central role in responses related to gendered ageism. In relation to ageing in place, older people prefer their habitual environment when they have moderate care needs and accept moves to nursing homes when their needs increase.

Mental Action Way of Understanding (WoU) and Way of Thinking (WoT) Students in Statistics Learning in Higher Education

This study aims to examine in depth the Mental Action Way of Understanding (WoU) and Way of Thinking (WoT) of students in learning Statistics in Higher Education and the relationship between WoU and WoT. This research is qualitative research using the Grounded Theory approach. The participants involved were 36 people in the city of Bandung, West Java, Indonesia. Validity and reliability using the Krippendorff content analysis method with the Inter-coder Agreement (ICA) test using Krippendorff alpha coefficients. The results showed that there was a significant relationship between students' mental action statistics and students' WoU and WoT in Statistical Understanding and Thinking. The relationship between WoU and WoT: (1) Students’ WoU will build such a statistical framework. (2) Students’ WoT affects how statistical modeling from their statistical knowledge. This relationship fits the philosophy of statistics and inductive inference. The statistical mental action of WoU students was to make models, analyze, calculate and generalize. The mental actions of Statistics students from WoT were interpreting, analyzing, predicting, reasoning, and inferring. The WoU and WoT mental action models developed the PPDC Cycle model and the Gal’s statistical literacy model. the PPDAC Cycle model by Wild and Pfankuch, consists of five dimensions, and one of the dimensions was the PPDAC Cycle. The statistical literacy model by Gal consists of two dimensions, namely the dimensions of statistical knowledge and disposition. The novelty of this research was developing the PPDAC Cycle statistical thinking model by Wild and Pfankuch, and the statistical literacy model on the statistical knowledge dimension by Gal. This study's results can be considered in designing statistical learning in universities and measuring students' understanding and statistical thinking skills.

Impacts of paediatric chronic pain on parents: A qualitative study.

Paediatric chronic pain adversely impacts the child's functioning, health-related quality of life and development. However, there is a need for a holistic assessment of parental impacts of caring for a child with chronic pain. This qualitative study aimed to investigate the possible psychosocial, functional and work impacts of caring for a child with chronic pain on parents, including any positive effects of the experience. Ten parents (eight mothers and two fathers), whose child attended the Sydney Children's Hospital Interdisciplinary Chronic and Complex Pain Clinic, participated in semi-structured interviews exploring the impact of caring for a child with chronic pain. Interviews were audio-recorded, transcribed, coded (with good inter-coder agreement) and analysed using thematic analysis. The parental experience of caring for a child with chronic pain was encapsulated by four overarching themes: (1) 'the constant and all-consuming nature of pain' - parents described the unpredictable, yet constant nature of chronic pain, contributing to wide-reaching impacts in various areas of their life; (2) 'dealing with uncertainty' - their experience was commonly characterized by a sense of uncertainty, stress, hopelessness and fear; (3) 'importance of support and self-care' - strong support networks and prioritizing self-care were crucial in alleviating the negative effects of paediatric chronic pain; and (4) 'a revitalized and optimistic view on life and relationships' - some parents identified unique and positive effects, such as stronger relationships, personal growth and a reformed view on life. This study provided rich data on the various impacts of caring for a child with chronic pain, highlighting the need for the development of holistic, family-centred interventions addressing both child and parental functioning.

Learning pediatric emergency medicine over time: A realist evaluation of a longitudinal pediatric emergency medicine clinical experience.

Emergency medicine (EM) practitioners must be proficient at caring for patients of all ages, including pediatric patients. Traditionally, EM trainees learn pediatric emergency medicine (PEM) through block rotations. This is problematic due to the seasonal nature of pediatric diseases and infrequent critical events. Spaced repetition learning theory suggests PEM would be better learned through longitudinal rotations. The transition to competency-based medical education (CBME) in Canada is accelerating the need to find novel ways to attain competencies in postgraduate training. At McMaster University, senior EM trainees can choose either traditional PEM blocks or longitudinal rotations. Our objective was to understand how learners experience these different rotations given the transition to CBME in Canada. Using a realist framework of program evaluation, we conducted semistructured interviews with key stakeholders (trainees, program directors, attending physicians) in EM. The realist framework was used to understand how context interacts with theoretical mechanisms to produce outcomes of interest. Data were analyzed using inductive, conventional content analysis. All investigators coded a subset of transcripts independently and in duplicate to achieve intercoder agreement. A total of 13 interviews were completed with trainees (n= 11) and staff physicians (n= 2). The learning experience exists within an educational and clinical context, which are logistically distinct but inseparable. The longitudinal learning experience appears to improve learning through spaced repetition, which prevents atrophy of skills and knowledge while also benefitting from the offsetting of seasonal variability associated with many pediatric diseases. Improved feedback and entrustment are facilitated through the building of coaching relationships over time. Barriers to the learning experience are related mainly to logistical difficulties associated with resolving longitudinal and blocked learning experiences. Improved relationships with the interprofessional team may provide distinct learning opportunities and improved team functioning. Block rotations were identified as more valuable to junior trainees learning fundamental concepts. Longitudinal learning provides numerous advantages to learning PEM, including increased case variety, spaced repetition of core concepts, and a perception of greater entrustment of the learner through formation of coaching relationships over time. Future projects looking to quantify the differences between longitudinal and block learning to objectively show a difference in skills and knowledge are needed.

A Systems Approach to Improving Foundational Reading Skills at a Preschool in India

Holistic Education School located in Mumbai, India, created a strategic improvement plan to address the poor foundational reading skills of its kindergarten students. Curriculum Based Measure (CBM) assessment data showed a rise in reading performance during the implementation of the plan, from 87% of the students scoring below grade level in 2016 to 89% achieving grade level benchmarks in 2021. An instrumental case study qualitative research design was employed to describe the nature of changes made in relation to the increase in performance. Data were collected from a variety of sources and coded through inductive and deductive processes that achieved intercoder agreement of 97.87% and 0.73 Kappa coefficient for reliability. Triangulation, member checking, and clarifying researcher bias, were employed to maximize validity. The findings describe the nature of transformation undertaken by the strategic improvement plan on key systems-level elements like curriculum, professional development, student engagement, parent engagement, and school leadership.

Inclusivity (Online Discussions/Discussion Quality)

Inclusivity (Online Discussions/Discussion Quality)

Interactivity/Reciprocity (Online Discussions/Discussion Quality)

Interactivity/Reciprocity (Online Discussions/Discussion Quality)

A closer examination of the STEM characteristics of the STEM activities published in NSTA journals

ABSTRACT Background Based on the literature review, shared characteristics of integrated STEM (I-STEM) curriculum includes an engaging and motivating context, student participation in engineering design activities, opportunity to redesign, engaging in mathematics or science content, having student-centred pedagogies, and teamwork and communication. These principles and more were used to analyse STEM activities published in NSTA journals. Purpose It was to determine the extent to which the STEM activities published in three NSTA journals (Science and Children, Science Scope, and The Science Teacher) between the years 2013 and 2020 reflected the shared characteristics of I-STEM education. Sample A total of 161 STEM teaching activities were analysed during the study. Methods A STEM rubric (abbreviated as RACSTEM) developed by the authors. Expert panel reviews and intercoder agreement calculations provided support for the content validity and reliability of RACSTEM. The rubric was composed of seven main categories, namely, integrating different disciplines, real-world problem, learner-centeredness, teamwork, communication, redesign, and assessment. STEM activities were analysed based on RACSTEM through content analysis and reported using descriptive statistics. Results Different combinations of STEM and non-STEM disciplines were integrated into the activities. 76% of the activities included real-world problems. Learner centred strategies (e.g. project-based learning) were utilized in 81% of the STEM papers. In 94% of the papers, students mostly participated in group work. Analysis revealed that 58% of the papers asked learners what they would change at the redesign stage. Finally, in the assessment aspect, the most problematic one, results showed discrepancies between the integrative nature of STEM and its assessment. Conclusion It was unclear how many I-STEM activities were published in NSTA publications that fit the requirements for I-STEM once the NGSS was launched. The findings of this study, as well as the development of the RACSTEM instrument, seem to have the ability to resolve ambiguity.