Observation Protocol Research Articles

Use of a nationwide, patient-powered observational platform to define the landscape of treatment, molecular profiling, and outcomes for fibrolamellar carcinoma.

e16192 Background: Fibrolamellar carcinoma (FLC) is a rare form of liver cancer, distinct from hepatocellular carcinoma, characterized by its prevalence among adolescents and young adults, capacity to form bulky tumors, and high rate of recurrence and metastasis. Without an established standard-of-care, the treatment of FLC is multidisciplinary, often involving surgery, radiation, and therapeutic interventions concurrently. Understanding the landscape of real-world treatments for FLC pts is imperative to defining an effective therapeutic approach. Methods: In partnership with the FibroFighters Foundation, we enrolled 102 patients (pts) in the observational research protocol XCELSIOR ( NCT03793088 ). Electronic medical records (EMR) were gathered from over 2,000 sites of care across 31 states - a median of 43 unique locations per pt - and utilized to populate a 21 CFR Part 11-compliant database with fully longitudinal, uninterrupted clinical data. Data elements were standardized to OMOP-based dictionaries. Anti-cancer medications were combined to define regimens and correlated with surgical dates. Pts were offered access to aggregated EMR and structured summaries as a digital personal health record via online portals. Results: Median age at diagnosis was 19 years (range 6-51 years) and median diagnosis date was 8/26/2020. At initial diagnosis, 36% of pts had distant metastases, but 86% of all pts eventually experienced metastatic disease with lung being the most frequent location (30 pts). The most prevalent anti-cancer therapies used were lenvatinib (75% pts exposed) and nivolumab (73% pts) and the most prevalent regimen was gemcitabine/lenvatinib/oxaliplatin (33% of pts). Among repurposed agents, 50% were exposed to quercetin, 44% to celecoxib, and 14% to metformin. 64% of pts underwent at least one surgical resection with 21% of these receiving neoadjuvant therapy including 7 distinct regimens, most commonly gemcitabine/lenvatinib/oxaliplatin. 91% of pts received adjuvant therapy for a median duration of 162 days. 22% of pts received radiation therapy and 15% received Y90 radioembolization. Among diagnostic profiling, 82% had somatic next generation sequencing, 48% had germline testing, 28.5% had organoid functional profiling, and 68% had Signatera testing performed. Conclusions: Patients with FLC undergo extensive molecular profiling with multiple novel therapeutic regimens being employed in real world practice. Aggregation of raw imaging scans is ongoing for central radiological review to measure depth and duration of response offered by the recorded therapeutic strategies. This dataset is a rich resource available to academic and government researchers to fuel research and define optimal treatment modalities. Topline statistics for the dataset are available at https://xcures.com/flcdashboard/ . Clinical trial information: NCT03793088 .

Observational study protocol for an arrhythmia notification feature

IntroductionScreening for atrial fibrillation (AF) in the general population may help identify individuals at risk, enabling further assessment of risk factors and institution of appropriate treatment. Algorithms deployed on wearable...

Computational Thinking Skills Theorization in The Vocational High School Computer Programming Subject Context

The purpose of the study is to compile concepts related to computational thinking skills in the context of computer programming subject. The research design used is data theorization (grounded theory) with open coding and axial coding data analysis techniques. Three types of data collection protocols are used: interview protocols, observation protocols, and recording protocols. The characteristics informants used are students who have experience learning computer programming in the vocational high school. The research produced several concepts, namely the problem concept, which then gave rise to the computational thinking skills concept. The problem concept consists of three sub-concepts: the common problem sub-concept, the medium problem sub-concept, and the high problem sub-concept. The concept of computational thinking skills is a set of skills required in the problem-solving process in computer programming. Computational thinking is not just a thought process but an expression of skills that others can observe. In the computer programming subject context, two concepts influence the process of mastering computational thinking skills: the guidance strategies concept and the information sources concept. The concept of success becomes a mediator in developing computational thinking skills. The theory produced in this study can be used as a basis for conducting research with quantitative designs related to deductive proof of the theory.

A Qualitative Study on Digital Citizenship Education in Primary Schools

In recent years, the number of studies related to digital citizenship has increased amidst the intense use of digital media. The literature, rich in theoretical and empirical examples related to this topic, provides a clear picture of contemporary society. There is little empirical research on digital citizenship education in the early school years. Thus, through this investigation, we aimed to collect qualitative data on the development of specific digital competencies and digital citizenship in primary school students. Therefore, the authors considered it appropriate to use the observation method in its participatory version. Over a year, a group of 29 pupils, aged between 6 and 7, formed the sample to identify how they would develop digital competencies and digital citizenship in the context of formal and non-formal learning experiences. To collect the necessary information, we used the observation protocol, conducted in the form of open-ended questions described by Gay, Mills & Airasian (2006). The results of the study showed that parental educational attainment, family structure, time spent online, and the type of device have an important impact on the development of digital competencies and digital citizenship in young learners.

Validation of the Nociception Level (NOLTM) Index for the detection of nociception and pain in critically ill adults: An Observational Study Protocol (Preprint)

Validation of the Nociception Level (NOLTM) Index for the detection of nociception and pain in critically ill adults: An Observational Study Protocol (Preprint)

Adventures with Anxiety: Gender bias in using a digital game for teaching vocational English

AbstractBackgroundIn Norway, games were introduced into the 2020 English curriculum; acknowledging games as language learning resources alongside other text‐based material. However, little is known about the type of games selected by English teachers, reasons for doing so, or how games are actually used in teaching. This is particularly relevant in gendered vocational classes, with mainly girls or boys respectively in different programmes, as national surveys show that while most teenage boys play digital games, comparatively few girls do.ObjectivesThe primary aim was to investigate how the digital commercial game Adventures with Anxiety was used during naturally occurring English teaching in four gendered vocational classes: one Technological and Industrial Production class (boys only) and three Healthcare, Childhood and Youth development classes (girls mainly).MethodsParticipants involved 36 students (aged 16–17) and two English teachers. Using a mixed methods design (Brevik, 2022), we combined quantitative and qualitative analyses of classroom video recordings (5.5 h), student screen recordings during gameplay (8 h) and retrospective teacher interviews (1.5 h). Analyses relied on the validated PLATO observation protocol (Grossman, 2015) and reflexive thematic analysis (Braun & Clarke, 2022).Results and ConclusionsEnglish teachers expressed a dual aim in using the game for teaching English; to improve students' language skills and provide experiences of how to live with anxiety. Although the game offered relevant vocabulary and virtual experiences with anxiety; anxiety emerged as a professional component of caring professions in the girls‐mainly classes, while being used to help boys acknowledge anxiety in their own lives.

Local Excision for organ preservation in early REctal cancer with No Adjuvant treatment (LORENA Trial): prospective observational study protocol

IntroductionLocal resection (LR) is an alternative to total mesorectal excision (TME) that avoids its associated morbidity to the detriment of oncological radicality in early stages of rectal cancer. There are several conditioning factors for the success of this strategy, such as poor prognosis histological factors (PPHF), involvement of resection margins, clinical under staging, or complications that may lead to the indication for radical surgery with TME. Patients and methodAn international multicenter prospective observational open-label study has been designed. Consecutive patients diagnosed with early rectal cancer (cT1N0 on MRI +/− endorectal ultrasound) whose lower limit is a maximum of 2 cm proximal to the ano-rectal junction will be included. The primary objective of the study is to determine the overall prevalence of PPHF after LR and requiring TME or postoperative radio-chemotherapy. DiscussionThe prevalence of PPHF conditioning the success of LR in early distal rectal cancer has been scarcely studied in the literature, and there are very few prospective data. Considering the increasing interest in the watch and wait strategy in rectal cancer and its possible application in early-stage tumors, it seems necessary to know this information.The results of this study will help guide clinical practice in patients with early distal rectal cancer. It will also provide quality information for the design of future comparative studies to improve organ preservation success in these patients.Trial registration number: NCT05927584.

Insider Views and Practices of Translanguaging Classrooms in Early Childhood Education

This study, situated within the domain of language education, delves into the intricate dynamics of translanguaging practices and their profound influence on learning and teaching procedures in a linguistically and culturally enriched early childhood education environment. Focused on the context of Greece, the research aims to uncover the nuanced ways in which translanguaging shapes the language communication and interaction of four-year-old children, considering both native Greek speakers and their peers from migrant or refugee backgrounds. The study cohort consists of 50 children aged four, comprising both native Greek and migrant-background children attending early childhood education. Employing a comprehensive research approach, the study includes in-depth, semi-structured interview sessions with young children and specific observational protocols designed to capture authentic interactions and communications. These methods are employed to trace the development of intercultural awareness and sensitivity within an environment conducive to translanguaging. The findings highlight that the incorporation of translanguaging practices in early education settings not only refines communication and intercultural skills among young children but also significantly influences cultural awareness. Moreover, translanguaging fosters active engagement in learning activities and provides educators with valuable insights and support in implementing innovative language teaching techniques. This research contributes to the advancement of language education practices, particularly within linguistically and culturally diverse early childhood education contexts, emphasizing the transformative impact of translanguaging on crucial aspects of language learning and teaching.

Seeing Through the Eyes of Classroom Observers: The Case of Rating Contrasted Groups of Lessons With Classroom Observation Measures

Classroom observations are commonly employed to assess quality of instruction in research and practice in mathematics education. However, there is more to be learned about how sensitive classroom observation protocols are to exemplars of strong mathematics instruction, and continuous refinements to observation protocols or rating processes that may be warranted. In this study, we use the public-released mathematics videos from the Third International Mathematics and Science Study (TIMSS) to examine how classroom observers, using two contemporary classroom observation instruments, rate a set of lessons whose instructional quality is in theory expected to differ, also referred to as contrasted groups. We find that descriptively, the pattern of findings is distinct from prior studies’ conclusions about the relative instructional quality reflected in the TIMSS video pool. We provide qualitative examples to illustrate the findings, and discuss implications for future research. We point to the potential value of exploring classroom observation rubrics’ performance using ‘contrasted groups’ of lesson videos, as a tool to broaden our understanding of how observation instruments are functioning.

An observational study protocol to capture, validate and characterise lucid episodes in people living with advanced dementia receiving hospice care

IntroductionLucid episodes (LEs) in advanced neurodegenerative disease, characterised by a transient recovery of abilities, have been reported across neurological conditions, including Alzheimer’s disease and related dementias. Evidence on LEs in...

Validity Evidence for an Observational Fidelity Measure to Inform Scale-Up of Evidence-Based Interventions.

As evidence-based interventions are scaled, fidelity of implementation, and thus effectiveness, often wanes. Validated fidelity measures can improve researchers' ability to attribute outcomes to the intervention and help practitioners feel more confident in implementing the intervention as intended. We aim to provide a model for the validation of fidelity observation protocols to guide future research studying evidence-based interventions scaled-up under real-world conditions. We describe a process to build evidence of validity for items within the Session Review Form, an observational tool measuring fidelity to interactive drug prevention programs such as the Botvin LifeSkills Training program. Following Kane's (2006) assumptions framework requiring that validity evidence be built across four areas (scoring, generalizability, extrapolation, and decision), confirmatory factor analysis supported the hypothesized two-factor structure measuring quality of delivery (seven items assessing how well the material is implemented) and participant responsiveness (three items evaluating how well the intervention is received), and measurement invariance tests suggested the structure held across grade level and schools serving different student populations. These findings provide some evidence supporting the extrapolation assumption, though additional research is warranted since a more complete overall depiction of the validity argument is needed to evaluate fidelity measures.

Care home staff interventions to optimise pain assessment and management in people with advanced dementia in long-term care settings: a systematic review

Abstract Introduction As a person’s dementia symptoms worsen, they become increasingly frail and dependent on others for support,[1] which often necessitates moving into long-term care (LTC). Pain is often poorly recognised and undertreated in people with advanced dementia.[2] Aim To identify and evaluate the effectiveness of care home staff interventions involving pain assessment and management in people with advanced dementia in LTC settings. Methods This systematic review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and registered with the International Prospective Register of Systematic Reviews (PROSPERO; CRD42022355487). Nine databases (Medline, Scopus, The Cochrane Library, EMBASE, Web of Science, International Pharmaceutical Abstracts, PsycINFO, CINAHL, World Health Organisation International Clinical Trials Registry Platform) were searched. The search strategy was developed in Medline and adapted as appropriate for the other databases; it comprised the following search terms: ‘dementia’, ‘Alzheimer’s disease’, ‘pain assessment’, ‘pain management’, ‘pain intervention’, ‘care home’, and ‘nursing home’. The selection of studies was limited to randomised controlled trials (RCTs) in which ≥50% of participants had advanced (moderate-severe) dementia and published in English. Electronic databases were searched from date of inception to May 2023. Two reviewers independently assessed potentially relevant articles for inclusion, completed data extraction and assessed risk of bias using the Cochrane Risk of Bias (RoB) 2.0 tool. A third reviewer was consulted when consensus could not be reached. A narrative analysis was undertaken due to the heterogeneity of outcome measures. Results In total, 3504 articles were identified, with 998 records remaining after duplicate removal. This left 2456 studies for title/abstract screening, and 22 full-text studies were assessed for eligibility; five studies met the inclusion criteria. A total of 1363 participants (mean age 83-88 years) and 123 care homes were included in the studies. The studies reported that these interventions were shown to be effective compared to control groups: the Pain recognition and Treatment (PRT) protocol in China, regular use of the PACSLAC pain assessment tool in Canada, the use of a Comprehensive Observational Pain Management Protocol in Hong Kong, the COSMOS intervention in Norway, and the STA OP stepwise multidisciplinary intervention in the Netherlands. Primary outcomes concerning pain scores measured by PACSLAC (n=2 studies), PAINAD (n=2 studies), and MOBID-2 (n=1 study) showed a statistically significant decrease. Secondary outcomes measured included several clinical outcomes associated with pain, depression, presence of neuropsychiatric symptoms and nursing staff stress. Four of the included studies were judged to be at ‘some concerns of bias’ according to RoB 2.0, due to unclear information in two or more domains (randomisation process, deviations from intended interventions, and measurement of the outcomes); one study was judged as having a ‘low risk of bias’. Conclusion The use of pain assessment tools decreased pain scores in people with advanced dementia in LTC. A limitation of this review is the exclusion of non-English language studies. Further, the clinical and methodological heterogeneity of included studies made a quantitative comparison difficult, thus, the findings are based on narrative analysis.

Integrating a Community-Based Health Information System with a Patient-Centered Medical Home to improve care of patients with hypertension: a longitudinal observational study protocol

BackgroundThe primary healthcare system in Pakistan focuses on providing episodic, disease-based care. Health care for low-middle income communities is largely through a fee-for-service model that ignores preventive and health-promotive services. The growing burden of cardiovascular illnesses requires restructuring of the primary health care system allowing a community-to-clinic model of care to improve patient- and community-level health indicators.MethodsWe propose a model that integrates a Patient-Centered Medical Home (PCMH) with a Community-Based Health Information System (CBHIS) using hypertension (HTN) as an example. This protocol describes the integration and evaluation of the PCMH-CBHIS infrastructure through a population-based, observational, longitudinal study in a low-middle income, urban community in Pakistan. Participants are being enrolled in CBHIS and will be followed longitudinally over two years for HTN outcomes. A mixed-methods approach is adopted to evaluate the process of integrating PCMH with CBHIS. This involves building partnerships with the community through formal and informal meetings, focus group discussions, and a household health assessment survey (HAS). Community members identified with HTN are linked to PCMH for disease management. A customized electronic medical record system links community-level data with patient-level data to track changes in disease burden. The RE-AIM evaluation framework will be used to monitor community and individual-level metrics to guide implementation assessment, the potential for generalization, and the effectiveness of the PCMH in improving HTN-related health outcomes. Ethical clearance has been obtained from the Ethics Review Committee at Aga Khan University (2022-6723-20985).DiscussionThis study will evaluate the value of restructuring the primary care health system by ensuring systematic community engagement and measurement of health indicators at the patient- and community-level. While HTN is being used as a prototype to generate evidence for the effectiveness of this model, findings from this initiative will be leveraged towards strengthening the management of other acute and chronic conditions in primary care settings. If effective, the model can be used in Pakistan and other LMICs and resource-limited settings.

Oxidative stress, redox status and surfactant metabolism in mechanically ventilated patients receiving different approaches to oxygen therapy (MecROX): An observational study protocol for mechanistic evaluation

Background MecROX is a mechanistic sub-study of the UK-ROX trial which was designed to evaluate the clinical and cost-effectiveness of a conservative approach to oxygen therapy for invasively ventilated adults in intensive care. This is based on the scientific rationale that excess oxygen is harmful. Epithelial cell damage with alveolar surfactant deficiency is characteristic of hyperoxic acute lung injury. Additionally, hyperoxaemia (excess blood oxygen levels) may exacerbate whole-body oxidative stress leading to cell death, autophagy, mitochondrial dysfunction, bioenergetic failure and multi-organ failure resulting in poor clinical outcomes. However, there is a lack of in-vivo human models evaluating the mechanisms that underpin oxygen-induced organ damage in mechanically ventilated patients. Aim The aim of the MecROX mechanistic sub-study is to assess lung surfactant composition and global systemic redox status to provide a mechanistic and complementary scientific rationale to the UK-ROX trial findings. The objectives are to quantify in-vivo surfactant composition, synthesis, and metabolism with markers of oxidative stress and systemic redox disequilibrium (as evidenced by alterations in the ‘reactive species interactome’) to differentiate between groups of conservative and usual oxygen targets. Methods and design After randomisation into the UK-ROX trial, 100 adult participants (50 in the conservative and 50 in usual care group) will be recruited at two trial sites. Blood and endotracheal samples will be taken at 0, 48 and 72 hours following an infusion of 3 mg/kg methyl-D9-choline chloride. This is a non-radioactive, stable isotope of choline (vitamin), which has been extensively used to study surfactant phospholipid kinetics in humans. This study will mechanistically evaluate the in-vivo surfactant synthesis and breakdown (by hydrolysis and oxidation), oxidative stress and redox disequilibrium from sequential plasma and bronchial samples using an array of analytical platforms. We will compare conservative and usual oxygenation groups according to the amount of oxygen administered. Trial registration: ISRCTN ISRCTN61929838, 27/03/2023 https://doi.org/10.1186/ISRCTN61929838.

Students’ performance, attitude, and classroom observation data to assess the effect of problem-based learning approach supplemented by YouTube videos in Ugandan classroom

In response to global demands, Uganda’s Vision 2040 seeks to transform the country into a modern and prosperous nation by implementing Sustainable Development Goal (SDG) 4, focusing on equitable and quality education. The 21st-century workforce requires individuals who can effectively navigate complex workplace challenges. This dataset was gathered from Form-2 Ugandan secondary school students (aged 12 to 15) across 12 schools in the Sheema District. The dataset comprises three types of data: students’ performance in a physics topic (simple machines), their attitudes toward problem-solving and critical thinking when learning physics using Problem-Based Learning (PBL) supplemented by YouTube videos, and classroom observations documented with the reformed teaching observational protocol (RTOP). The intervention of teaching using PBL was executed in 2022, collecting data from 973 lower secondary school students. The intervention involved three approaches: one group (144 students) received PBL along with YouTube videos, another group of 482 students received PBL alone, and a third group (347 students) was taught using the traditional method. This data article explains the study’s data creation, collection, and analysis process. The dataset holds significance for secondary school teachers, policymakers, and researchers, offering insights into the impact of PBL with and without ICT resources on learning physics and students’ attitudes toward these learner-centered approaches.

Stress and self-regulation behaviors in preterm neonates hospitalized at open-bay and single-family room Neonatal Intensive Care Unit

The Neonatal Intensive Care Unit (NICU) could be a risk factor for the development of preterm neonates due to the stressful procedures they undergo. Stress-related behaviors must be managed through environmental strategies that support regulating the neonates' biobehavioural system to minimize the negative impact on their development. The study aimed to compare the NICU environment's stressful procedures and developmental care strategies and the stress and self-regulation behaviors of preterm neonates in groups differentiated by the NICU environmental design. The sample comprised 20 preterm neonates hospitalized in a NICU with an open-bay model (OB NICU) and 20 preterm neonates hospitalized in a single-family room model (SFR NICU). The stressful procedures were assessed by the Neonatal Infant Stressor Scale (NISS). The developmental care strategies and the preterm neonates' stress and self-regulation behaviors were assessed using a structured observational protocol. The between-group comparison was performed by the Mann-Whitney test, and the significance level was set at 5%. Both NICUs had similar stressful procedures and developmental care approaches. However, the preterm neonates hospitalized in the SFR NICU exhibited significantly fewer total stress behaviors, and specifically in the motor system, compared to those in the OB NICU. Additionally, the preterm neonates hospitalized in the SFR NICU exhibited significantly more total self-regulation behaviors, and specifically in the behavioral state system, compared to those in the OB NICU. The findings showed that the single-family room NICU model was consistent with the environmental protection of biobehavioural regulation in preterm neonates hospitalized in the NICU.

Early rehabilitation after spinal cord injury: a scoping review protocol.

The objective of this scoping review is to identify the timings and types of early rehabilitation after spinal cord injury (SCI) that are documented in the literature, as well as identify possible research gaps in this field. Preclinical and clinical studies support the idea that early rehabilitation may be beneficial for patients with acute SCI. However, the timings and types of rehabilitation that should and could be used in the acute stage are still unclear. The first step towards such knowledge is to extensively review what is documented in the literature. This review will consider all papers focusing on early rehabilitation after SCI in adult humans or animals (<3 months' post-SCI for humans and other primates and <3 weeks' post-SCI for other animals). Interventions in the included papers must aim at sensorimotor or functional improvement and take place in a hospital or a rehabilitation center if they target human subjects. This review will include published and unpublished experimental and observational studies, research protocols, conference abstracts, systematic reviews, and guidelines. MEDLINE (PubMed), Scopus, Embase, PEDro, Web of Science Core Collection, CINAHL (EBSCOhost), Cochrane Database of Systematic Reviews, Cochrane CENTRAL, PROSPERO, ClinicalTrials.gov, and gray literature sources will be searched for eligible articles. No language or date limits will be applied. Two independent reviewers will perform study selection and data extraction, and the results will be presented according to the SCI type and severity. Open Science Framework https://osf.io/q45er.

Designing an observation protocol for professional development providers and mentors working with scientific inquiry‐supported classroom settings

AbstractSeveral observation protocols in different theoretical frameworks and components have been designed and validated by teacher trainers and professional development providers to capture and categorize observational data on the characteristics and level of inquiry in science practices. However, certain constraints limit their wide use, such as the neglect of certain indicators of scientific inquiry, the scoring of just summative goals, or the necessity for extensive observer training. The present study, therefore, aims to propose a tool that assists professional development providers and mentors in diagnosing the closeness of practices to scientific inquiry in the context of various settings, from traditional didactic lectures to scientific inquiry, making it easily operational for self and peer evaluations as well as 360‐degree video feedback. The method concentrated on multiple observations provides supportive evidence about the tool's internal consistency coefficient, the relationship between its components, and the inter‐rater reliability coefficient between multiple scorers. The results highlight that the tool, which includes the components of course structure, course overview, and teacher–student communication, has the potential to contribute to the growing library of observation protocols.

Feasibility and Acceptability of a Multifaceted Observational Protocol to Investigate Lucidity in Advanced Dementia.

Episodes of lucidity (ELs), characterized by spontaneous, transient recovery of abilities, are reported across neurological conditions, including advanced dementia. Despite the significance of these events, existing research is limited to retrospective reports. Approaches to prospectively capturing and characterizing ELs in dementia are lacking. This pilot study determined the feasibility and acceptability of a multifaceted observational protocol to capture, characterize, and validate ELs in individuals with advanced dementia in hospice. Feasibility was measured through enrollment/retention rates, workload/usability assessment of video observation, and data processing procedures to facilitate the measurement of verbal and nonverbal features for EL characterization. The acceptability of observation and informant validation procedures was qualitatively examined with clinician and family caregiver participants. Study procedures were endorsed as highly acceptable among clinician (N = 49) and caregiver (N = 16) participants, demonstrating higher than anticipated thresholds for observation duration. Enrollment and retention rates for patient participants (N = 6) were 100% and 84%, respectively. Workload and usability measures demonstrated low disruptiveness and high ease of use after training. Longitudinal video observation among individuals with advanced dementia in hospice care for the detection of lucidity was feasible and highly acceptable. Multidimensional, temporal coding of features of ELs is feasible but time-intensive.

Improving the Precision of Classroom Observation Scores Using a Multi-Rater and Multi-Timepoint Item Response Theory Model

ABSTRACT Classroom observational protocols, in which raters observe and score the quality of teachers’ instructional practices, are often used to evaluate teachers for consequential purposes despite evidence that scores from such protocols are frequently driven by factors, such as rater and temporal effects, that have little to do with teacher quality. In this study, we present an item response theory model that improves precision of observation scores by accounting for multiple sources of measurement error. Unlike the sum-scoring approach typically used in practice, the model presented here identifies raters and protocol items that are more reliable and should therefore be weighted more in scores. We then show that teachers identified as substandard using our model versus the sum-scoring approach are oftentimes very different. Our results suggest that accountability systems could improve observation scores by increasing the scoring weights of items and raters that provide greater information.