Reliability assessment of artificial intelligence autonomously generated diagnostics and treatment plans

Relevance. The objectivity of assessing the professional activities of pedagogical staff is a pressing task in modern education. This is due to dynamic changes in the organization of the educational process and increasing demands on the professional competence of pedagogical staff. Traditional methods of assessing the professional activities of pedagogical staff are often subjective and do not fully reveal their level of knowledge, skills, and abilities. In this context, artificial intelligence offers tools for a more accurate, transparent, and comprehensive assessment of the professional activities of pedagogical staff based on the analysis of data on teaching quality, their interaction with general secondary education students, and learning outcomes. Purpose. The purpose of the article is to investigate the role of artificial intelligence in enhancing the objectivity of assessing the professional activities of pedagogical staff in general secondary education institutions. Methods. The research methods included: studying scientific sources and regulatory documents concerning the use of artificial intelligence technologies in education to identify the state of research on the problem; theoretical analysis, synthesis, and generalization of views to substantiate the role of artificial intelligence in the objective assessment of the professional activities of pedagogical staff in general secondary education institutions; and generalization of findings. Results. The article substantiates the role of artificial intelligence in the objectivity of assessing the professional activities of pedagogical staff in general secondary education institutions based on the use of big data processing algorithms and comprehensive analytics, automation of the collection and analysis of quantitative and qualitative indicators of pedagogical staff's professional activities, the structure of their interaction with students, analysis of competency development dynamics, learning materials (through natural language processing), and student learning outcomes. The features of using artificial intelligence in assessing the professional activities of pedagogical staff are revealed through task personalization and the provision of individual recommendations for their further professional development. Key advantages and challenges associated with the use of artificial intelligence in assessing the professional activities of pedagogical staff in general secondary education institutions are identified. Conclusions. The study found that the use of artificial intelligence significantly enhances the objectivity, efficiency, and transparency of assessing the professional activities of pedagogical staff in general secondary education institutions, as it allows for a shift from quantitative assessment criteria to comprehensive analysis and contributes to the formation of individual trajectories for their professional growth. Successful implementation of artificial intelligence in assessing the professional activities of pedagogical staff is based on considering unified methodological approaches and standards, as well as the availability of appropriate technical infrastructure.

Focus issue: Artificial intelligence in medical physics.

Abstract: The application of Artificial Intelligence (AI) in learning assessment has attracted the attention of many educational experts, researchers and practitioners. This study discusses the opportunities and challenges of using AI in learning assessment. Traditional assessment has weaknesses in terms of misjudgment, inability to measure individual abilities that are not measured in certain forms of assessment, significant cost and time, slow feedback, and inability to be adjusted individually. Several studies have shown that the use of AI in assessments can improve the accuracy, validity and reliability of assessments, reduce human rater bias, enable adaptive assessments, increase time and cost efficiency, provide faster and more timely feedback, and assist in identifying individual needs and improve the quality of learning. However, the use of AI technology can only be a tool, and the final decision must still be made by humans. Therefore, the use of AI in assessment requires special attention in terms of ethics and the development of human capabilities to understand and use AI technology wisely.Keywords: Artificial Intelligence, Assessment Abstrak: Penerapan Artificial Intelligence (AI) dalam penilaian pembelajaran telah menarik perhatian banyak ahli pendidikan, peneliti, dan praktisi. Penelitian ini membahas peluang dan tantangan penggunaan AI dalam asesmen pembelajaran. Asesmen tradisional memiliki kelemahan dalam hal kesalahan penilaian, ketidakmampuan mengukur kemampuan individu yang tidak terukur dalam bentuk asesmen tertentu, biaya dan waktu yang signifikan, umpan balik yang lambat, dan ketidakmampuan untuk disesuaikan secara individual. Beberapa penelitian menunjukkan bahwa penggunaan AI dalam asesmen dapat meningkatkan akurasi, validitas, dan reliabilitas asesmen, mengurangi bias penilai manusia, memungkinkan asesmen adaptif, meningkatkan efisiensi waktu dan biaya, memberikan umpan balik yang lebih cepat dan tepat waktu, serta membantu dalam mengidentifikasi kebutuhan individu dan meningkatkan kualitas pembelajaran. Namun, penggunaan teknologi AI hanya dapat menjadi alat bantu, dan keputusan akhir tetap harus dilakukan oleh manusia. Oleh karena itu, penggunaan AI dalam asesmen memerlukan perhatian khusus dalam hal etika dan pengembangan kemampuan manusia dalam memahami dan memanfaatkan teknologi AI dengan bijak.Kata Kunci: Kecerdasan Buatan, Asesmen

There is emerging evidence which suggests the utility of artificial intelligence (AI) in the diagnostic assessment and pre-treatment evaluation of thyroid eye disease (TED). This scoping review aims to (1) identify the extent of the available evidence (2) provide an in-depth analysis of AI research methodology of the studies included in the review (3) Identify knowledge gaps pertaining to research in this area. This review was performed according to the 2020 Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (PRISMA). We quantify the diagnostic accuracy of AI models in the field of TED assessment and appraise the quality of these studies using the modified QUADAS-2 tool. A total of 13 studies were included in this review. The most common AI models used in these studies are convolutional neural networks (CNN). The majority of the studies compared algorithm performance against healthcare professionals. The overall risk of bias and applicability using the modified Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool led to most of the studies being classified as low risk, although higher deficiency was noted in the risk of bias in flow and timing. While the results of the review showed high diagnostic accuracy of the AI models in identifying features of TED relevant to disease assessment, deficiencies in study design causing study bias and compromising study applicability were noted. Moving forward, limitations and challenges inherent to machine learning should be addressed with improved standardized guidance around study design, reporting, and legislative framework.

This study aims to analyze the effectiveness of applying Artificial Intelligence (AI) in assessment gamification to improve student learning outcomes at SDN Kemayoran 2 Bangkalan. The research employed a quasi-experimental method with a nonequivalent control group design. The participants consisted of fourth-grade students divided into a control class and an experimental class. The instruments used included pretest and posttest assessments to measure students’ understanding before and after the intervention, as well as data analysis using an independent samples t-test, normality test, and N-Gain test. The study results indicated an increase in the average learning outcome scores for both the control group (81.68) and the experimental group (86.63), with the latter showing a significantly greater improvement. The independent samples t-test yielded a significance value of 0.00 (< 0.05), confirming data were accepted. Furthermore, the normality test indicated that the data were normally distributed, as the significance value was ≥ 0.05. Additionally, the N-Gain test yielded an average score of 0.80 (high improvement category) and an effectiveness percentage of 80.41% (highly effective category). Therefore, the study concludes that the application of Artificial Intelligence in assessment gamification is proven effective and has a substantial impact on improving student learning outcomes at SDN Kemayoran 2 Bangkalan. This research implies that AI-based educational technologies can serve as innovative alternatives to support the learning process in elementary schools.

NRG Oncology Assessment of Artificial Intelligence for Automatic Treatment Planning in Radiation Therapy Clinical Trials: Present and Future.

Introduction: The third molar is frequently affected in the oral cavity, with rare cases of inverted impaction in the maxillary region. This rarity poses unique challenges in diagnosis, treatment planning, and surgical procedures, with potential complications like sinusitis or infection. Current literature highlights a divide between conservative and surgical management, lacking comprehensive guidelines and exploring the role of AI-assisted tools. This study addresses this gap by evaluating the diagnostic accuracy of AI tools, particularly ChatGPT, against human specialists in Oral and Maxillofacial Surgery. Considering the growing role of AI in medicine, this research aims to provide insights into the potential of AI in enhancing diagnosis and treatment planning for rare cases, emphasizing collaboration between AI systems and medical professionals. Objectives: • Evaluate the diagnostic accuracy of AI tools (ChatGPT) compared to human-generated (specialist OMFS) diagnoses in dental cases. • Assess the efficiency and reliability of AI-assisted treatment plans in contrast to those generated by dental professionals. • Compare the performance and features of paid and non-paid versions of the AI programs utilized. Materials and Methods: This study centered on the unique case of a 59-year-old woman at Thumbay Dental Hospital, presenting issues related to a faulty dental bridge and a history of managed hypertension. An orthopantomogram showed the inverted impacted maxillary third molar Figure 1. The patient exhibited no direct symptoms from this impaction. A cone computed tomography was performed for a detailed analysis of the patient for complete prosthetic rehabilitation and academic purposes Figure 2. All the available data, including the history, clinical examination, and radiographic findings, were provided to a specialist and AI tools (ChatGPT version 3 and ChatGPT version 4) to get a diagnosis and treatment plan for this unusual case of an impacted third molar. Data collection comprised clinical examinations, imaging, and AI outputs, focusing on the accuracy of diagnostic and treatment plans. The study also assessed AI’s adaptability, cultural sensitivity, and practicality in clinical settings, aiming to gauge AI tools’ potential in enhancing dental diagnostics and treatment planning alongside human expertise. AI tools, including ChatGPT and its advanced versions, were employed to generate and compare diagnostic assessments and treatment plans against those created by dental professionals. Results: In a rare dental case involving a 59-year-old woman with a faulty dental bridge and managed hypertension, specialists at Thumbay Dental Hospital identified functional issues and an inverted impacted maxillary third molar using orthopantomogram and Cone Beam Computed Tomography. Collaborating with oral and maxillofacial surgeons, a comprehensive treatment plan for complete oral rehabilitation was formulated, considering age, anatomical complexity, and medical history, offering two options for the impacted third molar. The AI-generated diagnosis and treatment plans from ChatGPT versions 3 and 4 were explored. ChatGPT-3 provided a detailed plan for bridge replacement, including a specialized segment for managing the impacted third molar. ChatGPT-4 crafted a comprehensive plan starting with an initial consultation, encompassing diagnostic procedures, discussions on bridge replacement options, preparation, fabrication, fitting, and post-procedure care. The plan addressed missing teeth and the impacted tooth, highlighting adaptability to individual needs. However, ChatGPT-4 emphasized its inability to provide medical diagnoses, stressing the importance of professional evaluation. In summary, the study compares human-generated and AI-generated diagnosis and treatment plans. The human-generated plan prioritizes collaboration and comprehensive care, while AI-generated plans from ChatGPT versions 3 and 4 demonstrate detailed and adaptable approaches. ChatGPT-4 underscores the need for professional evaluation. The research sheds light on the potential roles of human and AI expertise in dental diagnostics and treatment planning, emphasizing the importance of collaboration for optimal patient care. Conclusion: This study highlights the collaborative potential of AI and human intelligence in handling intricate dental cases, such as Inverted Maxillary Third Molar Impaction. While AI tools like ChatGPT showcase the ability to create detailed treatment plans, their incapacity to replicate nuanced clinical judgment underscores the vital role of human oversight, particularly in specialized fields like Oral and Maxillofacial Surgery. The results are consistent with existing research, emphasizing AI as a supplement to, rather than a substitute for, human expertise in healthcare. The ongoing integration of AI with human medical practice shows promise in improving diagnostic accuracy and treatment effectiveness in dental healthcare.

This study focused on Ethical and Intellectual Considerations in the Deployment of Artificial Intelligence in Educational Assessment and Research among Social Studies Lecturers in Universities in south-south, Nigeria. Three research questions and 3 hypotheses guided the study. The study adopted a descriptive survey research design. The study comprised of 120 Social Studies lecturers in public federal and State universities in south-south, Nigeria. A structured questionnaire was used for data collection. Data were analyzed using descriptive statistics (mean and standard deviation) to summarize responses and t-test statistics to test the hypotheses. The findings revealed among others that there was significant difference in the level of awareness regarding the responsible and ethical use of AI technologies between Social Studies lecturers in Federal and State universities. The study concluded that although AI tools offer innovative prospects for educational advancement, there is a pressing need to enhance ethical awareness and address intellectual property challenges among Social Studies lecturers. It was recommended that universities and educational authorities should organize regular training workshops, develop clear ethical guidelines, and promote responsible use of AI tools in research and assessment

As artificial intelligence (AI) becomes more prevalent in day-to-day life, it is important to consider public opinion and acceptance towards these AI systems. Specifically, many struggle to trust AI when used to create medical treatment plans. After all, one’s health tends to be a very emotionally-charged issue and not necessarily what we would associate with a machine. To address this, we present the question: Do young college students from diverse backgrounds trust AI system-developed treatment plans? We hypothesized that participants would rate the treatment plan developed by the AI system lower than the treatment plan developed by a physician. We conducted a between-group randomized controlled experiment with 81 community college students (75% female, 25% male) from a Hispanic Serving Institution. We presented the control group with a case study in which a physician designed the treatment plan. We presented the experimental group with a case study in which an AI system designed the treatment plan. The AI-developed treatment plan scored lower on the trust rating scale than the physician-created treatment plan, which is consistent with the hypothesis. There was no statistically significant difference between the two groups' scores on the Healthcare Trust Questionnaire. Our results also showed no significant difference between the trust levels in AI of people of different ages, genders, ethnicities, employment statuses, or hospitalization statuses, contradicting previous research. Overall, our findings may indicate a negative public opinion regarding AI-developed treatment plans, potentially deterring the future of AI- driven healthcare.

BackgroundStroke-induced upper limb dysfunction requires functional assessment and rehabilitation. The intelligent rehabilitation assessment and virtual reality training system for upper limb motor function in stroke can accurately and objectively assess patients’ motor function and guide their rehabilitation training. Our study aims to verify the clinical efficacy of the virtual reality training system in improving upper limb motor dysfunction in poststroke patients.Methods/DesignThis study will be a single-center, single-blind, randomized controlled clinical study. Fifty eligible patients will be randomized in a 1:1 ratio into a virtual reality training group (VR) and a conventional upper extremity treatment (CT) group. The intervention will be performed five times per week for 4 weeks. The primary outcome will be the Fugl-Meyer Motor Function Assessment—Upper Extremity (FMA-UE), and the secondary effects will be kinematic and electromyographic assessments. Adverse events will be recorded, and serious adverse events will be used as criteria for discontinuation of the intervention.DiscussionA stroke upper limb motor function assessment and virtual reality rehabilitation training system based on the FTHUE scale can achieve a close link between intelligent assessment and treatment of upper limb motor function in poststroke patients while integrating the design concepts of the upper limb and hand assessment and treatment, which can theoretically improve upper limb function in stroke to a greater extent, but further high-quality studies are needed. The results of this trial will determine whether an assessment and training system based on the FTHUE scale can improve upper extremity motor dysfunction after stroke.Trial registrationChinese Clinical Trial Registration Center, ChiCTR2200060214. Registered May 22, 2022. Manuscript Version: 2.0 Manuscript Date: May 2, 2025.

Achieving highly efficient treatment planning in intensity-modulated radiotherapy (IMRT) is challenging due to the complex interactions between radiation beams and the human body. The introduction of artificial intelligence (AI) has automated treatment planning, significantly improving efficiency. However, existing automatic treatment planning agents often rely on supervised or unsupervised AI models that require large datasets of high-quality patient data for training. Additionally, these networks are generally not universally applicable across patient cases from different institutions and can be vulnerable to adversarial attacks. Deep reinforcement learning (DRL), which mimics the trial-and-error process used by human planners, offers a promising new approach to address these challenges. PURPOSE: This work aims to develop a stochastic policy-based DRL agent for automatic treatment planning that facilitates effective training with limited datasets, universal applicability across diverse patient datasets, and robust performance under adversarial attacks. We employ an actor-critic with experience replay (ACER) architecture to develop the automatic treatment planning agent. This agent operates the treatment planning system (TPS) for inverse treatment planning by automatically tuning treatment planning parameters (TPPs). We use prostate cancer IMRT patient cases as our testbed, which includes one target and two organs at risk (OARs), along with 18 discrete TPP tuning actions. The network takes dose-volume histograms (DVHs) as input and outputs a policy for effective TPP tuning, accompanied by an evaluation function for that policy. Training utilizes DVHs from treatment plans generated by an in-house TPS under randomized TPPs for a single patient case, with validation conducted on two other independent cases. Both online asynchronous learning and offline, sample-efficient experience replay methods are employed to update the network parameters. After training, six groups, comprising more than 300 initial treatment plans drawn from three datasets, were used for testing. These groups have beam and anatomical configurations distinct from those of the training case. The ProKnow scoring system for prostate cancer IMRT, with a maximum score of 9, is used to evaluate plan quality. The robustness of the network is further assessed through adversarial attacks using the fast gradient sign method (FGSM). Despite being trained on treatment plans from a single patient case, the network converges efficiently when validated on two independent cases. For testing performance, the mean standard deviation of the plan scores across all test cases before ACER-based treatment planning is . After implementing ACER-based treatment planning, of the cases achieve a perfect score of 9, with only scoring between 8 and 9, and no cases being below 7. The corresponding mean standard deviation is . This performance highlights the ACER agent's high generality across patient data from various sources. Further analysis indicates that the ACER agent effectively prioritizes leading reasonable TPP tuning actions over obviously unsuitable ones by several orders of magnitude, showing its efficacy. Additionally, results from FGSM attacks demonstrate that the ACER-based agent remains comparatively robust against various levels ofperturbation. We successfully trained a DRL agent using the ACER technique for high-quality treatment planning in prostate cancer IMRT. It achieves high generality across diverse patient datasets and exhibits high robustness against adversarialattacks.

Background: Navya is a validated online cancer informatics solution that combines artificial intelligence (AI) based analysis of the guidelines and evidence, and rapid review (2 mins/case) by organ specific tumor board experts at Academic Medical Centers to deliver multidisciplinary expert treatment plan to patients within 24 hours. Initially developed for bresat cancer patients in India without ready access to expertise, over 28,000 patients across 68 countries with all cancers have since reached out to Navya. Prior research (SABCS 2014-2018 and ASCO 2017) showed, 1) 97% concordance of Navya predictions with an academic medical center in India and in the US 2) 97% of patients experienced significant anxiety relief due to the rapid, 24 hours turnaround time at the time of making a critical decision. 3) 79% of patients received treatment concordant with Navya recommendations on the ground. Unlike synchronized 1 patient: 1 doctor virtual consults in telemedicine where multidisciplinary collaborations are difficult, Navya uses AI to summarize medical cases and predict treatment plans that can be rapidly modified/vetted by multidisciplinary experts in 1-2 minutes asynchronously and collaboratively on a mobile app. This scales access to expertise for patients around the world beyond the limited availability of experts' time for telemedicine and in-person consults. Methods: Three patient centered outcomes (travel distance, cost and time to receive expert treatment plan) were studied. All consecutive breast cancer patients who reached out to Navya between 1/1/17-1/31/19 but ultimately opted for in-person visit to an academic medical center were contacted by prospective phone follow up. This was compared to a balanced random sample of patients who only used Navya to obtain treatment plans. Results: 195 in-person patients were reached for a prospective phone follow-ups and 132 of them had completed their visit at the academic medical center. 335 Navya patients were analyzed in the control group. The groups did not differ significantly in demographics or disease characteristics. In-person patients and Navya patients differed significantly with respect to 1) median travel distance (838 miles, IQR (237 -1105 miles) vs. 0 miles (p < 0.05)) 2) travel related costs of $1597 [95% CI +/- $240] vs $105 online processing fee 3) total time to receipt of treatment plan (7.23 days, IQR (0.45 - 22.35 days) vs. 0.51 days IQR (0.19-1.057) (p < 0.05)). Conclusions: Cancer informatics solutions like Navya leverage AI to summarize the case, and predict guidelines and evidence based options. Combined by design with expert vetting from academic medical centers, such solutions can generate multidisciplinary treatment plans tailored to an individual patient. This scales ready access to expertise around the world. For patients with limited access to academic medical centers, such solutions eliminate travel burden, and significantly reduce cost and wait time to obtain a treatment plan. For experts who have no time to engage in telephone, video or written remote consults, vetting recommendations from the AI system with concise case summaries only takes 1-2 minutes per case. This model has shown significant ability to create access to specialty expertise, reduce the movement of patients to obtain treatment opinions, save costs and tremendously reduce waiting time for expertise in breast cancer, globally. Citation Format: Rajendra Badwe, Benjamin Anderson, Nancy Feldman, Sudeep Gupta, Shona Nag, CS Pramesh. Prospective comparison of cost, travel burden, and time to obtain multidisciplinary tumor board treatment plan through in-person visits vs. an AI enabled health technology [abstract]. In: Proceedings of the 2019 San Antonio Breast Cancer Symposium; 2019 Dec 10-14; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2020;80(4 Suppl):Abstract nr P6-13-04.

Leveraging Artificial Intelligence (AI) Clinical Decision Support Software to Improve Treatment Plan Quality in Head and Neck Cancer Patients

The article focuses on ways to use artificial intelligence in assessment and enhancement of creativity. This topic seems very important in the context of the intensive development of computer technologies providing people with the vast range of opportunities to improve their professional skills and intensify their personal development. Some particular ways of the use of artificial intelligence are analyzed. Artificial intelligence can operate independently and generate its own creative ideas. At the same time, it can interact with humans within the creative process or serve as a “creative assistance” of humans. The results of the empirical studies in this area showed that the efficacy of the artificial intelligence in the course of assessment and enhancement of human creativity is determined to considerable extent by a task given, a particular area which artificial intelligence operates in, and the specific forms of its interactions with humans. In some areas (e.g., generation of alternative uses), artificial intelligence can outperform humans, whereas in other tasks (e.g., creative writing) humans perform better that artificial intelligence. Some practical recommendations on how to optimize the use of artificial intelligence in assessment and enhancement of creativity, were proposed. Results of the study can be used in the development of creativity assessment methods as well as for the improvement of interaction between people and artificial intelligence.

Artificial intelligence (AI) has become a prominent focus in orthodontics. This study aimed to compare treatment plans generated by AI platforms (ChatGPT, Google Bard, Microsoft Bing) with those formulated by an experienced orthodontist. This observational cross-sectional pilot study aims to evaluate the effectiveness of AI-powered platforms in creating orthodontic treatment plans, using a clinical case treated by an experienced orthodontist as a benchmark. A clinical case was selected, and after obtaining informed consent, detailed case information was presented to ChatGPT-3.5, Microsoft Bing Copilot, and Google Bard Gemini for treatment planning. The AI-generated plans, along with the orthodontist's plan, were evaluated by 34 orthodontists using a questionnaire that included Likert scale and Visual Analog Scale (VAS) items. Statistical analysis was performed to compare the levels of agreement with the proposed treatment plans. Orthodontists exhibited significantly higher levels of agreement with treatment plans proposed by the orthodontist, compared to those generated by AIs platforms (p < 0.001). Both Likert scale and VAS scores indicated increased confidence in the orthodontist's expertise in formulating treatment plans. No significant differences were found among the AI platforms, though Google Bard received the lowest mean scores. Orthodontists demonstrated a higher level of acceptance of treatment plans formulated by human counterparts over those generated by AI platforms. While AI offers significant contributions, the clinical judgment and experience of orthodontists remain essential for thorough and effective treatment planning in orthodontics.