External Evaluation Research Articles

External evaluation of a commercial artificial intelligence-augmented digital auscultation platform in valvular heart disease detection using echocardiography as reference standard

ObjectiveThere are few studies evaluating the accuracy of commercially available AI-powered digital auscultation platforms in detecting valvular heart disease (VHD). Therefore, the utility of these systems for diagnosing clinically significant VHD remains unclear. We conducted a comprehensive external evaluation of the Eko murmur analysis software (EMAS) and report its accuracy in detecting murmurs associated with VHD using echocardiography (ECHO) as the reference standard. MethodsWe analyzed phonocardiogram (PCG) and ECHO data from 1,029 individuals (461 females, mean (SD) age: 61 (29) years, BMI: 29 (9)) at a single academic medical center. PCGs were recorded using the EkoDUO and EkoCORE stethoscopes from the four standard auscultation positions immediately before transthoracic ECHO (TTE) testing. TTE diagnostics were used as reference to calculate the EMAS sensitivity and specificity in detecting murmurs associated with VHD. The 95% confidence intervals are reported. ResultsOf the 4,081 PCGs, 79% were of sufficient quality for murmur analysis. The sensitivity and specificity of the EMAS in detecting VHD were 39.3% (95% CI: 37.2–41.3) and 82.3% (95% CI: 80.0–84.5), respectively. EMAS sensitivity in detecting murmurs associated with common VHD types was 62.5%, 75.0%, 88.9%, and 63.3% for moderate-severe and severe cases of mitral stenosis, aortic regurgitation, aortic stenosis, and mitral regurgitation, respectively. ConclusionThe EMAS algorithm exhibits limited overall sensitivity in detecting VHD. The sensitivity of the algorithm varies across VHD types. These findings suggest that EMAS can be used for diagnosis of specific lesions, but not all VHD types, which limits its clinical applicability as a screening tool.

Leveraging the Kirkpatrick four‐level model to evaluate evaluation capacity building work

AbstractIn this article, we reflect on a decade of using the Kirkpatrick four‐level model to evaluate a multifaceted evaluation capacity building (ECB) initiative. Traditionally used to assess business training efforts, the Kirkpatrick model encourages evidence to be gathered at four levels: reaction, learning, behavior, and results. We adapted these levels to fit the context and information needs of the EvaluATE project, an ECB initiative funded by the National Science Foundation. As members of the external evaluation and project teams, throughout the article we describe how each level was modified and translated into evaluation questions. Our adapted Kirkpatrick levels are implementation and reach, satisfaction, learning, application, and impact. Using these adapted Kirkpatrick levels to ground our evaluation challenged us to integrate multiple data sources to tell a comprehensive story that served the information needs of the project team and the funder. Overall, we found the Kirkpatrick model to be practical, accessible, and flexible, allowing us to capture the multidimensional aspects of the ECB initiative. However, there are opportunities to enhance the utility of the Kirkpatrick framework by integrating other evaluation approaches, such as culturally responsive and equitable evaluation and principles‐focused evaluation.

Using a novel PSMA-PET and PSA-based model to enhance the diagnostic accuracy for clinically significant prostate cancer and avoid unnecessary biopsy in men with PI-RADS ≤ 3 MRI.

The diagnostic evaluation of men with suspected prostate cancer (PCa) yet inconclusive MRI (PI-RADS ≤ 3) presents a common clinical challenge. [68Ga]Ga-labelled prostate-specific membrane antigen ([68Ga]Ga-PSMA) positron emission tomography/computed tomography (PET/CT) has shown promise in identifying clinically significant PCa (csPCa). We aim to establish a diagnostic model incorporating PSMA-PET to enhance the diagnostic process of csPCa in PI-RADS ≤ 3 men. This study retrospective included 151 men with clinical suspicion of PCa and PI-RADS ≤ 3 MRI. All men underwent [68Ga]Ga-PSMA PET/CT scans and ultrasound/MRI/PET fusion-guided biopsies. csPCa was defined as Grade Group ≥ 2. PRIMARY-scores from PSMA-PET scans were evaluated. A diagnostic model incorporating PSMA-PET and prostate-specific antigen (PSA)-derived parameters was developed. The discriminative performance and clinical utility were compared with conventional methods. Internal validation was conducted using a fivefold cross-validation with 1000 iterations. In this PI-RADS ≤ 3 cohort, areas-under-the-curve (AUCs) for detecting csPCa were 0.796 (95%CI, 0.738-0.853), 0.851 (95%CI, 0.783-0.918) and 0.806 (95%CI, 0.742-0.870) for PRIMARY-score, SUVmax and routine clinical PSMA-PET assessment, respectively. The diagnostic model comprising PRIMARY-score, SUVmax and serum free PSA/total PSA (fPSA/tPSA) achieved a significantly higher AUC of 0.906 (95%CI, 0.851-0.961) compared to strategies based on PRIMARY-score or SUVmax (P < 0.05) and markedly superior to conventional strategies typically based on PSA density (P < 0.001). The average fivefold cross-validated AUC with 1000 iterations was 0.878 (95%CI, 0.820-0.954). Theoretically, using a threshold of 21.6%, the model could have prevented 78% of unnecessary biopsies while missing only 7.8% of csPCa cases in this cohort. A novel diagnostic model incorporating PSMA-PET derived metrics-PRIMARY-score and SUVmax-along with serum fPSA/tPSA, has been developed and validated. The integrated model may assist clinical decision-making with enhanced diagnostic accuracy over the individual conventional metrics. It has great potential to reduce unnecessary biopsies for men with PI-RADS ≤ 3 MRI results and warrants further prospective and external evaluations.

Legal regulation of education in Ukraine. Part 6: Features of admission to higher education in Ukraine

This article examines the legal issues of regulating the access of citizens of Ukraine to higher education of the third educational and scientific degree (doctor of philosophy) in the paradigm of international acts, the Constitution of Ukraine, the Laws of Ukraine, and Resolutions of the Cabinet of Ministers of Ukraine in the field of education. This is related to the existence of certain legal conflicts in the education sector, which arose as a result of the ambiguity and imperfection of the legislation of Ukraine on education, which the Ministry of Education and Science of Ukraine uses to illegally restrict the access of citizens of Ukraine to higher education of the third educational and scientific degree. Taking into account the requirements of the Constitution of Ukraine, the Universal Declaration of Human Rights of the United Nations, and the Convention for the Protection of Human Rights and Fundamental Freedoms, the article presents the results of a study of the peculiarities of the application of the provisions of the relevant legislative acts in the field of education (with all changes and additions introduced as of September 1, 2024), namely the Law of Ukraine “On Education” (dated September 5, 2017, under No. 2145-VIII), the Law of Ukraine “On Higher Education” (dated July 1, 2014, under No. 1556- VII) and the “Procedure for Training Candidates for Higher Education Degrees of Doctor of Philosophy and Doctor of Sciences in institutions of higher education (scientific institutions)”, approved by Resolution No. 261 of the Cabinet of Ministers of Ukraine (dated March 23, 2016). At the same time, the research is mainly focused on the peculiarities of applying the above­mentioned legislative acts to determine the differences in the conditions of access to free and paid education and the established legitimacy and feasibility of certain elements of such conditions. Principally this concerns the introduction by the State of Ukraine of unified professional exams and other tests (with the use of external independent evaluation) as a condition for access to the next stage of tests, namely, entrance exams and other tests for admission to study introduced by higher education institutions of Ukraine. That is, the legality of establishing a two-level system of competitions and entrance tests for those who wish to obtain a higher education of the third educational and scientific degree (doctor of philosophy) on a paid basis (i.e., at their own expense or the expense of non-state grants and scholarships) is being considered. The article is the sixth one within the cycle of studies in which the peculiarities of access of Ukrainian citizens to higher education in martial law conditions are addressed.

External evaluation of Brain Injury Guideline (BIG) low risk criteria for traumatic brain injury

BackgroundFewer than 20 % of traumatic brain injury (TBI) cases with traumatic intracranial hemorrhage (ICH) result in clinical deterioration. The Brain Injury Guideline (BIG) criteria were published in 2014 and categorize patients with TBI into three risk groups (BIG 1, 2, and 3) based on CT scan findings, neurological examination, anti-coagulant/platelet medications, and intoxication. Early data is promising, suggesting no instances of neurosurgical intervention or death in the low-risk BIG1 category within 30 days. We sought to externally validate the BIG criteria and identify patients with TBI at low risk of clinical deterioration. We hypothesized that patients meeting the BIG1 low risk criteria have less than a 1 % risk of death or neurosurgical intervention. MethodsWe performed a retrospective cohort study of a level 1 trauma center's trauma registry records from 2011 to 2022 to identify patients with head trauma. We abstracted demographics, injury characteristics, clinical course, CT imaging results, and outcomes, and we categorized patients according to the BIG criteria. The Clopper-Pearson Exact method was used to estimate outcome frequency with confidence intervals. The primary outcome was death or neurosurgical intervention within 30 days. Secondary outcomes included progression on repeat head CT (RHCT), ICU admission with neurocritical care intervention, and TBI-related hospital readmission within 30 days. ResultsA total of 1714 patients with TBI with ICH were identified from the trauma registry. 325 patients were excluded due to missing data, pregnancy, incarceration, polytrauma, or GCS < 13, leaving 1389 for analysis. 193 patients (13.9 %) were classified as BIG1. No patients classified as BIG1 experienced the primary outcome measures of death or neurosurgical intervention (95 % confidence interval [CI]: 0 %–1.9 %). The number of patients who experienced the secondary outcome measures of progression on RHCT, ICU admission with neurocritical care intervention, or TBI-related hospital readmission within 30 days were 9 (4.7 %, 95 % CI: 2.2 %–8.7 %), 1 (0.5 %, 95 % CI: 0 %–2.9 %), and 4 (2.1 %, 95 % CI: 0.6 %–5.2 %), respectively. ConclusionBIG1 criteria identified a low-risk subset of patients with TBI with ICH. However, an upper 95 % CI of 1.9 % does not exclude the risk of neurologic deterioration being <1 %. Validation of these criteria in larger cohorts is warranted.

Recommendations of the Queensland children's gender service external evaluation and their implications for health policy in Australia.

To review the evaluation process and the implications for Australian health policy of the 2024 external clinical service evaluation of the Queensland Children's Gender Service (QCGS) and its recommendations. Failing to perform a systematic review of the relevant literature, and adhering to discredited and ideologically based guidelines, the Review made recommendations lacking evidentiary support that have major implications for Queensland and other Australian health services. The evaluation report's recommendations reveal eight areas of concern about the clinic's functioning.

Internalizing External Accident Costs in Safety Investment Evaluation Using Cost–Benefit Analysis

The primary objectives of transportation system development in European countries include improving transport efficiency, enhancing national economic competitiveness, and promoting sustainable mobility. A key aspect of achieving these goals is the evaluation of transport externalities, particularly external accident costs, which is critical for ensuring the economic and social sustainability of road transport development. This study comprehensively evaluates external road accident costs and proposes an innovative approach to internalizing these costs through a cost–benefit analysis (CBA). By integrating these external costs into CBA, policymakers can better understand the societal impact of road safety investments, allowing for more informed decisions regarding measures to reduce road accidents and fatalities. This paper also explores the role of specific safety investments, such as installing vehicle restrain systems (VRS), buffers, and terminals on the national road network, as well as installing speed cameras, in reducing accidents and mitigating external costs. The findings highlight road safety measures’ importance in enhancing economic efficiency and social well-being, providing a solid foundation for the sustainable and efficient development of transportation systems.

Marketing Strategy for Mitraguna Online Financing Products in Increasing the Number of Customers

This study aims to determine potential marketing strategies to increase the number of customers in the online mitraguna financing product of Bank Syariah Indonesia, Kisaran sub-branch office. This research uses qualitative research methodology. In qualitative research, the findings are analyzed and then reported by the researcher. In this study, two different types of data were used, namely primary and secondary. Interviews were used to obtain primary data, while publications from relevant financial institutions as well as relevant scientific literature were used to collect secondary data. The results of the data analysis resulted in the largest threat weight of 0.092538, the largest opportunity weight of 0.571429, the highest weakness weight of 0.210526, and the highest strength weight of 0.421053. The average overall score of the Internal Factor Evaluation matrix is 2.736842, but the average score of the External Factor Evaluation matrix is 2.952381. The results of this study indicate that the marketing strategy that needs to be implemented lies in quadrant V, namely the strategy of maintaining and maintaining by obtaining 5 alternative strategies that are expected to contribute to the Bank Syariah Indonesia Kisaran sub-branch office to increase the number of online mitraguna financing customers.

The influence of message frame and product type on green consumer purchase decisions : an ERPs study

Green consumption is a crucial pathway towards achieving global sustainability goals. Product-oriented green advertisements can effectively stimulate consumers’ latent needs and convert them into eventual purchasing intentions and behaviors, thereby promoting green consumption. Given that neuromarketing methods facilitate the understanding of consumers’ decision-making processes, this study combines prospect theory and need fulfillment theory, employing event-related potentials (ERPs) as measures to explore changes in consumers’ cognitive resources and emotional arousal levels when confronted with green products and advertising information. This enables inference regarding consumers’ acceptance of purchasing and their psychological processes. Behavioral results indicated that message framing influences consumers’ purchases, with consumers consuming more green in response to negatively framed advertisements. EEG results indicated that matching positive framing with utilitarian green products was effective in increasing consumers’ cognitive attention in the early cognitive stage. In the late stage of cognition negative frames stimulated consumers’ mood swings more, and the influence of product type depended on the role of message frames, and the consumption motivation induced by the product, whose influence was overridden by external evaluations such as message frames. These research findings provide an explanation for the impact of frame information on consumers’ purchasing decisions at different stages, assisting marketers in devising diverse promotional strategies based on product characteristics to foster the development and practice of green consumption. This will further embed the concept of green consumption advocated by organizations such as the United Nations Environment Programme (UNEP), World Wildlife Fund (WWF), and Greenpeace into the public consciousness.

Matrix Support: interface between Specialty Dental Centers and primary oral health care in Brazil

BackgroundMatrix Support (MS) is a strategy that can be used to improve integration between Primary Health Care (PHC) and other levels of care.ObjectiveThe aim of this study was to investigate the association between MS carried out in Brazilian Dental Specialty Centers (CEOs) (secondary level of oral health care) and aspects of the integrated work process with PHC, as well as contextual variables.MethodsA quantitative cross-sectional study was conducted using data from the Program for Quality Improvement and Access to CEOs (PMAQ-CEO). Secondary data from the External Evaluation of the second cycle of PMAQ-CEO were analyzed, including contextual variables obtained from sources such as the Unified Health System (SUS) and official research institutions. Descriptive analyses were performed, and four multiple models were adjusted to investigate the association between the variables.ResultsThe results showed that about half of the CEOs did not carry out therapeutic projects developed with the oral health teams of PHC. It was found that the lack of therapeutic projects developed with the teams was associated with the lack of discussion of complex cases by the team, lack of discussion of individual therapeutic project, absence of joint continuing education activities, lack of construction and discussion of clinical protocols, and lack of belief in the importance of planning and periodic evaluation. The results suggest that the articulation between PHC and secondary oral health care still presents weaknesses within the scope of SUS. Comprehensive care needs to be strengthened, requiring greater intervention from management.ConclusionIt was concluded that the individual factors of CEOs, related to the work process, have a greater influence on the lack of integration with oral health teams of PHC, compared to the contextual variables of municipalities.

Development of international guidelines by Tuina practitioners for specific acupoints of paediatrics Tuina (2022 version).

To establish a standardized framework encompassing the precise locations, manipulations, functions and indications of specific acupoints in the field of paediatric Tuina. The development of consensus involved three distinct stages. Initially, a list of paediatric Tuina specific acupoints was compiled based on an extensive literature review, which was subsequently supplemented through expert interviews. In the second stage, the Delphi method was employed to assess the significance of acupoint locations, manipulations, functions, and indications. In situations where the questionnaire survey failed to yield agreement or when the experts held reservations, the nominal group approach was utilized during the expert consensus meeting. The final version of the technical standardized material was ultimately determined during an expert consensus conference. After undergoing external peer review and evaluation, the completed draft was prepared for public dissemination RESULTS: The comprehensive list identified a total of 66 specific acupoints. The location and manipulation questionnaire consisted of 156 items based on the literature database, while the function and indication questionnaire contained 116 items. Two rounds of Delphi surveys were conducted for the location and manipulation category, and another two rounds of Delphi surveys were conducted for the function and indication category. During the experts consensus meeting The panel of experts conducted in-depth discussions on 61 questions, resulting in the formulation of technical guidelines for the locations, manipulations, functions, and indications of 64 paediatric Tuina acupoints. Subsequently, the research team compiled and edited the draft of the technical guidelines for acupoints of paediatric Tuina, which was finalized after external review and feedback. This study successfully established the recognized technical standards for practitioners of paediatric Tuina, thereby standardizing clinical practices and providing a foundation setting the framework for future research. The guidelines offer theoretical insights and recommendations for conducting clinical studies comparing different acupoint sites, as well as modifying or enhancing treatment regimens.

Prospective and External Evaluation of an Artificial Intelligence Model for Continuous and Early Prediction of Moderate and Severe AKI in Critically Ill Patients

Prospective and External Evaluation of an Artificial Intelligence Model for Continuous and Early Prediction of Moderate and Severe AKI in Critically Ill Patients

Deep Learning Virtual Contrast-Enhanced T1 Mapping for Contrast-Free Myocardial Extracellular Volume Assessment.

The acquisition of contrast-enhanced T1 maps to calculate extracellular volume (ECV) requires contrast agent administration and is time consuming. This study investigates generative adversarial networks for contrast-free, virtual extracellular volume (vECV) by generating virtual contrast-enhanced T1 maps. This retrospective study includes 2518 registered native and contrast-enhanced T1 maps from 1000 patients who underwent cardiovascular magnetic resonance at 1.5 Tesla. Recent hematocrit values of 123 patients (hold-out test) and 96 patients from a different institution (external evaluation) allowed for calculation of conventional ECV. A generative adversarial network was trained to generate virtual contrast-enhanced T1 maps from native T1 maps for vECV creation. Mean and SD of the difference per patient (ΔECV) were calculated and compared by permutation of the 2-sided t test with 10 000 resamples. For ECV and vECV, differences in area under the receiver operating characteristic curve (AUC) for discriminating hold-out test patients with normal cardiovascular magnetic resonance versus myocarditis or amyloidosis were tested with Delong's test. ECV and vECV showed a high agreement in patients with myocarditis (ΔECV: hold-out test, 2.0%±1.5%; external evaluation, 1.9%±1.7%) and normal cardiovascular magnetic resonance (ΔECV: hold-out test, 1.9%±1.4%; external evaluation, 1.5%±1.2%), but variations in amyloidosis were higher (ΔECV: hold-out test, 6.2%±6.0%; external evaluation, 15.5%±6.4%). In the hold-out test, ECV and vECV had a comparable AUC for the diagnosis of myocarditis (ECV AUC, 0.77 versus vECV AUC, 0.76; P=0.76) and amyloidosis (ECV AUC, 0.99 versus vECV AUC, 0.96; P=0.52). Generation of vECV on the basis of native T1 maps is feasible. Multicenter training data are required to further enhance generalizability of vECV in amyloidosis.

Analisis Strategis Rukita Winter

Rukita as a boarding house service provider is one of the main players in fulfilling the housing needs of students, workers, and the general public. With the tight competition in the boarding house market, especially in metropolitan areas such as West Jakarta, Rukita Winter has its own challenges to ensure its excellence and competitiveness. Successfully attracting the attention of boarding house tenants requires the right action in business strategy. Therefore, Rukita Winter always strives to adjust its business strategy to meet the needs and preferences of boarding house tenants. This study aims to identify and analyze the business strategies implemented by Rukita Winter. In addition, this study also aims to analyze the strengths (Strengths), weaknesses (Weaknesses), opportunities (Opportunities), and threats (Threats) faced by Rukita Winter's business. The method used involves the use of the IFE (Internal Factor Evaluation), EFE (External Factor Evaluation), IE (Internal-External), and QSPM (Quantitative Strategic Planning Matrix) matrices to analyze Rukita Winter's position. This study is expected to contribute to increasing knowledge and understanding of SWOT analysis and how its application can help Rukita Winter in formulating business strategies.

Selecting the Best Pharmacokinetic Models for a Priori Model-Informed Precision Dosing with Model Ensembling.

When utilizing population pharmacokinetic (popPK) models for a priori dosage individualization, selecting the best model is crucial to obtain adequate doses. We developed and evaluated several model-selection and ensembling methods, using external evaluation on the basis of therapeutic drug monitoring (TDM) samples to identify the best (set of) models per patient for a priori dosage individualization. PK data and models describing both hospitalized patients (n = 134) receiving continuous vancomycin (26 models) and patients (n = 92) receiving imatinib in an outpatient setting (12 models) are included. Target attainment of four model-selection methods was compared with standard dosing: the best model based on external validation, uninformed model ensembling, model ensembling using a weighting scheme on the basis of covariate-stratified external evaluation, and model selection using covariates in decision trees that were subsequently ensembled. Overall, the use of PK models improved the proportion of patients exposed to concentrations within the therapeutic window for both cohorts. Relative improvement of proportion on target for best model, unweighted, weighted, and decision trees were - 7.0%, 2.3%, 11.4%, and 37.0% (vancomycin method-development); 23.2%, 7.9%, 15.6%, and, 77.2% (vancomycin validation); 40.7%, 50.0%, 59.5%, and 59.5% (imatinib method-development); and 19.0%, 28.5%, 38.0%, and 23.8% (imatinib validation), respectively. The best (set of) models per patient for a priori dosage individualization can be identified using a relatively small set of TDM samples as external evaluation. Adequately performing popPK models were identified while also excluding poor-performing models. Dose recommendations resulted in more patients within the therapeutic range for both vancomycin and imatinib. Prospective validation is necessary before clinical implementation.

Site selection for a high-level nuclear waste repository in Germany: a short guide

The government agency BGE started a site selection process for a high-level nuclear waste repository in 2017. The first phase of data screening was completed in 2020 and resulted in the identification of search areas covering 54% of the national territory. The actual second phase will be completed by 2027 or later, and the areas for aboveground investigations will be identified. The obligatory nongovernment input is provided by follow-up regional conferences, which are best supported by external scientific expertise. The large amount of text documents and maps and the interactive internet tools such a 3D viewer provide a suitable basis for external evaluation. The rationale is to apply stricter criteria than those used in the first phase to drastically reduce the size of the search area. With this updated set of criteria, the size of the search area can be reduced to 2,662 km2 or 0.7% of the national territory. This size is convenient for detailed aboveground investigations such as high-density 3D seismic surveys.

ZNO-Eval: Benchmarking reasoning capabilities of large language models in Ukrainian

As the usage of large language models for problems outside of simple text understanding or generation increases, assessing their abilities and limitations becomes crucial. While significant progress has been made in this area over the last few years, most research has focused on benchmarking English, leaving other languages underexplored. This makes evaluating the reasoning and robustness level of language models in Ukrainian particularly challenging. The purpose of this work is to establish a comprehensive benchmark for the reasoning capabilities evaluation of large language models in the Ukrainian language. This paper presents the ZNO-Eval benchmark based on real exam tasks from Ukraine's standardized educational testing system: the External Independent Evaluation and the National Multi-subject Test. With single- answer options, multiple-choice, matching, and open-ended questions from diverse subjects, including Ukrainian language, mathematics, history, and geography, this dataset paves the way toward a thorough analysis of reasoning capabilities across different domains and complexities. Evaluation of several well-known language models, such as GPT-3.5-Turbo, GPT-4o, GPT-4-Turbo, Mistral Large, Claude 3 Opus, and Gemini-1.5 Pro on this benchmark demonstrated the superiority of GPT-4o in both common knowledge reasoning and intricate language tasks. At the same time, Gemini Pro and GPT-4 Turbo excelled in the arithmetic domain, leading in single-answer and open-ended math problems. While all models were close to max performance in text-only common knowledge tasks like history and geography, there still is a gap for Ukrainian language and math, thus highlighting the importance of developing specialized language benchmarks for more accurate assessments of model capabilities and limitations across different languages and contexts. This research introduced ZNO-Eval, an effective benchmark for evaluating reasoning capabilities, and thoroughly explored the abilities and limitations of modern solutions in the Ukrainian language. Future research should aim to expand the scope of ZNO-Eval to other modalities like images commonly used for exam problem description.

Can you mitigate gender bias in student evaluations of teaching? Evaluating alternative methods of soliciting feedback

Student evaluations of teaching can be unreliable indicators of effective teaching and affected by implicit bias. We conduct a randomized experiment at a selective U.S. liberal arts college in which we vary both the instrument and timing at which we solicit student feedback to assess whether either intervention can mitigate gender disparities in qualitative evaluation comments. External evaluators with expertise in teaching evaluation scored the positivity, specificity, and constructiveness of student comments in control and treatment groups. We find neither intervention is successful in mitigating gender disparities. Students that receive an alternate prompt which highlights the potential for implicit bias and asks them to articulate and apply their own criteria for effective teaching penalize female faculty at similar rates as those receiving a more standard evaluation prompt. We also find no evidence that delaying solicitation of student feedback until the start of the next semester, when students face less stress and fatigue, reduces the gender gap in the positivity of feedback. Instead, we find substantial evidence that the positivity of qualitative student evaluations is affected by student, instructor, and course characteristics across all study conditions. The results suggest that bias in student evaluations may be difficult to mitigate.

Abstract A021: Development of cross-ancestry polygenic risk. scores for breast cancer in women of African ancestry

Abstract Introduction: Polygenic risk scores (PRSs) predict breast cancer (BC) risk and have shown utility for screening in European ancestry women (EUR). However, PRSs developed for non-European ancestry, particularly African American (AA) women, have limited predictive accuracy. Methods: We analyzed data from 17,454 BC cases and 19,093 controls in the African American Breast Cancer Genetics (AABCG) consortium, split into training (70%), tuning (10%), and testing (20%) sets. Genome-wide association analyses for overall BC, ER-positive BC, ER- negative BC and triple-negative BC (TNBC) were performed in the training dataset, controlling for age, genotyping platform, and the top 10 principal components. We built single-ancestry PRS models using forward stepwise selection (FSS), PRSice-2, PLINK C+T, Lassosum2, and LDpred2-auto. These were combined with published EUR 313-SNP models (330-SNP for TNBC) in the tuning set to obtain cross-ancestry models. Genome-wide cross-ancestry PRS methods, including CT-SLEB and PRScsx, were also used. We created an ensemble model by combining different PRS models across subtypes using ensemble-based approaches. All developed PRS models were then evaluated in the independent testing set. Additional validation was performed using the All of Us (AoU) study with 834 BC cases and 29,122 controls from the AA population. Results: The observed-scale SNP-based heritability was estimated at 0.29 (SE: 0.12) for overall BC risk, 0.43 (SE: 0.18) for ER-positive BC, 0.44 (SE: 0.24) for ER-negative BC, and 1.13 (SE: 0.40) for TNBC. The best models showed good discriminating capacity: ensemble-based models for overall (AUC=0.626) and ER-positive BC (AUC=0.626), and combined cross-subtype and cross-ancestry FSS models for ER-negative (AUC=0.647) and TNBC (AUC=0.654). The odds ratio per standard deviation was 1.63 (95% CI: 1.55, 1.71) for the ensemble-based PRS of overall BC risk, with an estimated life risk of 28% for the top 1%. The cross-ancestry model using CT-SLEB combined with the EUR 313-SNP model was validated (AUC=0.61) among AA women in the AoU study. Conclusion: We developed PRSs with improved discriminating ability for BC risk in women with African ancestry. Cross- ancestry and subtype learning enhanced model performance. These findings indicate common variation has a larger role in BC in AA women than previously reported, warranting further validation in external AA populations and clinical evaluation. Citation Format: James L. Li, Haoyu Zhang, Julian C McClellan, Guimin Gao, Kevin Wang, Julie R. Palmer, Olufunmilayo I. Olopade, Christopher A. Haiman, Dezheng Huo. Development of cross-ancestry polygenic risk. scores for breast cancer in women of African ancestry [abstract]. In: Proceedings of the 17th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2024 Sep 21-24; Los Angeles, CA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2024;33(9 Suppl):Abstract nr A021.

CURRENT STATE OF TRAINING PROGRAM EVALUATION ACCORDING TO VIETNAMESE STANDARDS - SOLUTIONS TO ENSURE TRAINING PROGRAM QUALITY

The evaluation of training programs is usually conducted in at least two stages: self-assessment and external assessment, based on domestic and international standards. The evaluation results according to these standards, especially Vietnam's set of standards, serve as a foundation for developing improvement solutions to enhance the quality of training programs. The research team selected a method based on the statistical analysis of external evaluation results of training programs according to the standards issued in Circular 04/2016/TT-BGDĐT by the Ministry of Education and Training. From there, they forecast and manage existing issues as a basis for proposing 17 groups of quality assurance solutions for training programs, meeting the minimum requirements of stakeholders. Keywords: Quality; training program; external assessment; education quality assessment.