Quality Of Questions Research Articles

Artificial Intelligence as a Discriminator of Competence in Urological Training: Are we there?

Assessments in medical education play a central role in evaluating trainees' progress and eventual competence. Generative artificial intelligence (AI) is finding an increasing role in clinical care and medical education. The objective of this study is to evaluate the ability of the large language model ChatGPT to generate exam questions that are discriminating in the evaluation of graduating urology residents. Graduating urology residents representing all Canadian training programs gather yearly for a mock exam that simulates their upcoming board certifying exam. The exam consists of a written multiple-choice questions (MCQs) exam, and an oral OSCE. In 2023, ChatGPT Version 4 was used to generate 20 MCQs that were added to the written component. ChatGPT was asked to use Campbell-Walsh Urology, AUA, and CUA guidelines as resources. Psychometric analysis of the ChatGPT MCQs was conducted. The MCQs were also researched by 3 faculty for face validity and to ascertain if they came from a valid source. The mean score of the 35 exam takers on the ChatGPT MCQs was 60.7% versus 61.1% for the overall exam. 25% of ChatGPT MCQs showed a discriminating index > 0.3, the threshold for questions that properly discriminate between high and low exam performers. 25% of ChatGPT MCQs showed a point biserial > 0.2, which is considered a high correlation with overall performance on the exam. The assessment by faculty found that ChatGPT MCQs often provided incomplete information in the stem, provided multiple potentially correct answers, were sometimes not rooted in the literature. 35% of the MCQs generated by ChatGPT provided wrong answers to stems. Despite what appears to be similar performance on ChatGPT MCQs and the overall exam, ChatGPT MCQs tend not to be highly discriminating. Poorly phrased questions with potential for AI hallucinations are ever present. Careful vetting for quality of ChatGPT questions should be undertaken before their use on assessments in urology training exams.

LearningViz: a dashboard for visualizing, analyzing and closing learning performance gaps—a case study approach

The availability of large-scale learning data presents unprecedented opportunities for investigating student learning processes. However, it is challenging for instructors to fully make sense of this data and effectively support their teaching practices. This study introduces LearningViz, an interactive learning analytics dashboard to help instructors identify, analyze, and close performance gaps among students in their classes. In this dashboard, we incorporated three modules to enhance human and computer interactions for better supporting the teaching practices: the Student Overall Performance Analysis Module, which provides a comprehensive understanding of students’ learning in the course; the Student Group Performance Analysis Module, which examines performance gaps across different groups and identifies factors contributing to these gaps; and the Final Exam Item Analysis Module, which evaluates the quality of exam questions and identifies strategies for closing performance gaps. The overall design of the platform follows a user-centered approach, integrating data analysis with various visualization strategies in a unified platform. A case study is then conducted to highlight the effectiveness of LearningViz in supporting instructors analyzing students’ learning patterns and associated factors impacting learning performance. We further conduct a usability test with several domain experts, to evaluate the usefulness and effectiveness of this platform in supporting the teaching practices. Our findings underscore the platform's ability to support instructors in detecting performance gaps among students, investigating influential factors, evaluating assessment quality and implementing targeted instructional strategies for closing performance gaps.

Using artificial intelligence to provide a ‘flipped assessment’ approach to medical education learning opportunities

Purpose of the article Generative AI can potentially streamline the creation of practice exam questions. This study sought to evaluate medical students’ confidence using generative AI for this purpose, and overall attitudes towards its use. Materials and methods The study used a mixed-methods approach with a pre-post intervention design. 68 medical and physician associate students were recruited to attend a workshop where they were shown how to use Google Bard (now Gemini) to write exam questions before being encouraged to do this themselves with guidance. A survey was completed before and after. Seven students also participated in a follow-up focus group. Results The results showed an increase in participants’ confidence in using AI to write practice exam questions (p < 0.001) after the workshop. Qualitative feedback highlighted pros and cons of using generative AI to write exam questions, alongside some concerns about its implementation. Students noted other positive uses in the curriculum and expressed a desire for institutional clarity on appropriate AI use. Conclusions While increased confidence is positive, rigorous evaluation of AI-generated question quality is needed to confirm accuracy. Teaching students to use generative AI to create and critique practice questions represents a means of encouraging appropriate AI use.

Efficient generation of HPLC and FTIR data for quality assessment using time series generation model: a case study on Tibetan medicine Shilajit.

The scarcity and preciousness of plateau characteristic medicinal plants pose a significant challenge in obtaining sufficient quantities of experimental samples for quality evaluation. Insufficient sample sizes often lead to ambiguous and questionable quality assessments and suboptimal performance in pattern recognition. Shilajit, a popular Tibetan medicine, is harvested from high altitudes above 2000m, making it difficult to obtain. Additionally, the complex geographical environment results in low uniformity of Shilajit quality. To address these challenges, this study employed a deep learning model, time vector quantization variational auto- encoder (TimeVQVAE), to generate data matrices based on chromatographic and spectral for different grades of Shilajit, thereby increasing in the amount of data. Partial least squares discriminant analysis (PLS-DA) was used to identify three grades of Shilajit samples based on original, generated, and combined data. Compared with the originally generated high performance liquid chromatography (HPLC) and Fourier transform infrared spectroscopy (FTIR) data, the data generated by TimeVQVAE effectively preserved the chemical profile. In the test set, the average matrices for HPLC, FTIR, and combined data increased by 32.2%, 15.9%, and 23.0%, respectively. On the real test data, the PLS-DA model's classification accuracy initially reached a maximum of 0.7905. However, after incorporating TimeVQVAE-generated data, the accuracy significantly improved, reaching 0.9442 in the test set. Additionally, the PLS-DA model trained with the fused data showed enhanced stability. This study offers a novel and effective approach for researching medicinal materials with small sample sizes, and addresses the limitations of improving model performance through data augmentation strategies.

Protecting the protectors: redefining immunity protections for National Human Rights Institutions

ABSTRACT National Human Rights Institutions (NHRIs) established pursuant to the United Nations Paris Principles play a crucial role in protecting human rights at the domestic level, but they face significant risks of interference and obstruction when carrying out their sensitive functions. International standards recommend that NHRIs’ enabling legislation includes functional immunity provisions to protect them from these risks to their independence and effectiveness. However, presenting the results of the first comprehensive analysis of the functional immunity protections for NHRIs across 111 countries, this article finds that 40% of NHRI laws lacked any such protection provisions, while the remaining provisions were of questionable quality. To protect NHRIs from interference and obstruction, this article proposes a redefinition and strengthening of NHRI functional immunity provisions based on an analysis of existing protection provisions, international recommendations, and comparable immunity protections for judges and parliamentarians. It proposes a new framework for functional immunity that aims to grant NHRI leadership and staff the necessary level of protection against interference and obstruction, which also has wider relevance for other similar domestic bodies.

Current Concepts in Orthobiologics for Achilles Tendon Injuries: A Critical Analysis Review.

» Platelet-rich plasma and hyaluronic acid are low-risk and potentially high-reward treatments for Achilles tendinopathy, although clinical studies have yielded mixed results with questionable methodological quality» Case series and reports have reported that bone marrow aspirate, stem cells, and amniotic membrane products can improve functional outcomes, alleviate pain, and facilitate return to sport and activities, but high-level evidence studies are lacking» Exosomes are a promising novel biologic with laboratory studies showing improved collagen organization and cell proliferation, greater tendon mechanical properties, and prevention of extracellular matrix breakdown.» Standardization of protocols with clear reporting is necessary for future studies evaluating orthobiologic therapies for Achilles tendon injuries.

Arabic Question Generation Using Transformers

After the increased reliance on online education, online assessment became an essential tool for educators to remotely monitor and evaluate students’ understanding in order to assist them properly. However, the laborious process of creating exam questions is a challenge for most teachers. Thus, automated Question Generation aims to assist teachers by generating questions from given data. Limited research has been conducted to tackle this issue in the Arabic Language due to the complexity of the language and the limited amount of available Arabic data. This paper explores different implementations of the transformer models, that demonstrated their superiority in natural language processing. Three approaches were introduced to tackle this problem with Arabic data using an Arabic-based transformer, an English-based transformer, and a multilingual-based transformer. Each of the fine-tuned models was trained using ARCD, XGLUE, DialectBench, and ArabicQA data sets and evaluated on automatic and manual metrics. Two of the proposed models achieve state-of-the-art results on the Arabic question generation task. The English transformer obtained a ROUGE score of 0.59 on XGLUE, while the Arabic transformer model achieves 0.49 on ARCD. Both of these models demonstrate excellent quality of questions through human-conducted evaluations by achieving low WER and high GC, U, and A scores.

Implementation of Test Item Analysis for Learning Evaluation at SMAS YKPP Dumai

The activity of analyzing items is an activity that must be carried out by the teacher, this is done to evaluate the quality of the test questions used during the exam. So far, teachers have used the anbuso excel application to complete the analysis. The existing application requires entering student answer data and question answer keys one by one, this allows data errors when entering student answer data, and can be imagined if the teacher teaches more than one class it will take more time to be able to complete the analysis and increase the teacher's workload. This research aims to create a question analysis application with the codeigneter framework to help teachers get question analysis results quickly and efficiently without the need to enter student answers one by one. The application created presents analysis data based on the level of difficulty, differentiating power and function of the examiner which is made in tabular form so that it is easy to read. The application provides several users such as teachers, students and admins with different menu access rights. This application allows teachers to enter or change questions along with the answer key, view answer results and get question analysis. Students can get the results of the exam answers available after the exam. For admins, it helps in managing user data, monitoring exam students and activating exams. From the results of the analysis produced by the application, the teacher can use the question criteria.

Towards digitalization of the circular economy in the furniture industry

Fashionable, cheap furniture of questionable quality is often used for short-term economic gain. If the furniture is not liked, it is quickly discarded. The need to reduce CO2 emissions, dwindling resources and social and political demands contradict this practice and require the implementation of closed-loops material cycles and the introduction of circular business models. A systematic literature review and a case study were conducted to provide a comprehensive overview of current circular economy strategies (e.g. reuse, refurbishment and recycling) implemented in the furniture industry in relation to various key activities of platform companies. As part of the case study, interviews were conducted with stakeholders and experts from the furniture and interior design industry. The European Circular Economy Stakeholder Platform was also consulted. The literature review and the case study show that the furniture industry is in a state of transition. By incorporating designs that allow for improved remanufacturing, refurbishment and reuse, companies in the industry are increasingly starting to consider circular economy patterns. Increasing digitization and servitization are driving new circular business models. Another key finding is that digital platforms are suitable for connecting stakeholders such as manufacturers, service providers and consumers, taking into account the relevant business models. The study also identified motivations, barriers and requirements for the implementation of circular and digital business models in the furniture and interior design industry. There is still a large gap between what can be achieved with a circular economy and its actual implementation.

Enhancing exam question quality in medical education through bootstrapping.

Medical schools are required to assess and evaluate their curricula and to develop exam questions with strong reliability and validity evidence, often based on data derived from statistically small samples of medical students. Achieving a large enough sample to reliably and validly evaluate courses, assessments, and exam questions would require extensive data collection over many years, which is inefficient, especially in the fast-changing educational environment of medical schools. This article demonstrates how advanced quantitative methods, such as bootstrapping, can provide reliable data by resampling a single dataset to create many simulated samples. This economic approach, among others, allows for the creation of confidence intervals and, consequently, the accurate evaluation of exam questions as well as broader course and curriculum assessments. Bootstrapping offers a robust alternative to traditional methods, improving the psychometric quality of exam questions, and contributing to fair and valid assessments in medical education.

Water tariffs and social equity: Towards water service connections and pricing instruments for the poor

AbstractAside from operational sustainability and efficiency goals, water tariffs also need to address the requirements of affordability and equity, in most instances, by providing subsidies to poor households. Yet, poor households are unable to enjoy the benefits of such cross‐subsidization and linked conveniences as they are unable to get water service connections. Even with a well‐designed water tariff, the inability of poor households to get piped connections results in the payment of premiums for water that is difficult to access and, at times, of questionable quality. In this paper, we assess the pro‐poor projects of three Philippine municipal water systems and suggest that providing water service connections to poor households allows these households to avoid paying the water poverty premium, as well as experience the related public health and quality of life benefits accruing from such connections. We argue that designing the best tariff for existing socio‐economic, political, and environmental realities must go hand in hand with programs that provide water service connections to poor households as this will ensure not only operational viability but also equitable water provision.

Assessing of Biology Test Question Quality Using Anates Software

This research uses a quantitative descriptive approach. This research aims to determine the quality and appropriateness of Class XI Biology question items consisting of validity, reliability, level of difficulty, distinguishing power and distractor quality. The subjects are SMA N 1 Driyorejo Gresik as many as 30 students. Data collection in this research was carried out through student answer sheets, then the data was processed using Anates Ver 4.0.9. The test results show that the questions are appropriate and of good quality with a correlation of 0.62 and a reality coefficient of 0.76. This resulted in the question criteria being declared as having high validity, with four levels of difficulty and a small number of the quality of the distractor has poor quality test items.

Leveraging Qualitative Analysis Software to Complement Traditional Qualitative Research Approaches for Curriculum Design: Results of a Needs Assessment Study

ABSTRACT Educational programs are continually challenged with maintaining a curriculum that meets the current and future needs of the industry they serve, and accreditation requirements. Research methodologies of interviews, focus groups, and surveys to accomplish these requirements have a long-standing documented history in the literature. Interviews and focus groups can provide collection of rich information, but have limited sample size, and greater potential for participant and researcher bias. Survey research can be more objective and reach larger population, but results are limited to the type and quality of questions posed by the researchers, and honesty of the participant responses. Technological advances in data collection and analysis have been improving research methodologies for decades. This paper describes a research approach for conducting a program needs assessment. Using structured focus groups and qualitative artificial intelligence software, MAXQDA, the methodology identified 15 themes for activities commonly performed by entry-level construction management graduates. Within the themes, 17 common types of knowledge, and 15 types of skills were identified. Key knowledge included Terms & Symbols, Project Documents, Materials & Methods, Software Platforms, Scopes of Work, and Contract Structures. The top skills identified were Oral & Written Communication, Using/Learning Software/Technology, Collaboration/Teamwork, Analyzing/Interpreting Documents, and Personal Organization/Time Management.

User experience of the Written Exam Question Quality tool to inform the writing of new written-exam questions

Background: Creating new written-exam questions is a burdensome task for faculty members. While several guidelines exist, there had not been a previous attempt to streamline them in a user-friendly tool. We created the Written Exam Question Quality tool (WEQQ) and explored potential users’ perception of this tool when writing their exam questions. Methods: We conducted a descriptive study to explore how four Canadian faculty members used the WEQQ. We conducted structured interviews that were analyzed within and across participants to understand the latter’s perceived usefulness and acceptability of the WEQQ. Quantitative data from a short questionnaire on creating exam questions and their psychometric properties were also collected. Results and conclusion: Participants’ perception of the WEQQ was positive, and they were favorable to its use. The WEQQ seemed to represent a user-friendly, easy way to help faculty members in creating multiple-choice or short-answer questions. Time on task remained the same when using the WEQQ. We were able to identify two user profiles, passive and active, which indicated how faculty members use the WEQQ to create exam questions. Future steps would be to further investigate if the WEQQ can increase the quality of written-exam questions and to understand how to promote an active use of the WEQQ when implementing this tool.

ITEMAN-Based Evaluation of End-of-Semester Assessment Items: A Case Study of Language Test in Indonesian School Context

Assessment is an inseparable part of the learning process. Assessment is a way to decide the choices that a teacher will make for students, so that what they do is truly measurable. In line with this description, this study aims to determine the quality of the end-of-semester assessment questions for Indonesian, grade VIII, at SMPN 8 Yogyakarta. The method used in this research is descriptive quantitative. Quantitative descriptive is used to analyze documents in the form of output analysis results from the ITEMAN application. The results of the review that will be submitted later are in the form of validity, reliability, IDB, ITK and the functioning of the distractor. In obtaining the results of the study. The research steps carried out in this study were are testing the validity of the question grids, testing the reliability, IDB, ITK, and distractors through the ITEMAN program, analyze the results according to expert opinions and research that are relevant to this study, and conclude the results of the study. There were 50 questions that were analyzed and tested on 32 class VIII students of SMPN 8 Yogyakarta. Overall, the research results show all questions are declared valid, Alpha is 0.926. This shows that the reliability of the items is very high, IDB results show 38 items in good, 5 moderate, 4 sufficient, and 3 failed categories, the results of ITK analysis showed 31 very easy questions, 10 easy questions, 5 moderate questions, 2 difficult questions, and 2 very difficult questions, and the results of the distractor function show that 32 questions do not work, and 18 item distractors work. Overall, the results of this analysis aim to provide a reference base related to the quality of questions through ITEMAN, especially in learning Indonesian, which so far has not been done much

Communication Quality and the Cost of Language: Evidence from Stack Overflow

The transmission of information is crucial for productivity and growth. However, language differences may limit its effectiveness. This is particularly relevant for knowledge platforms that aim to be global, given the cultural diversity of the pool of users. In this paper, I empirically investigate how the exogenous cost of language affects communication quality and the trade-offs faced by knowledge platforms in implementing their website in multiple languages. I exploit the staggered introduction of websites for languages other than English on a question-and-answer platform to demonstrate two main facts. First, nonnative English speakers who contribute in English increase their answers’ quality once able to use their native language, and their answers are more likely to solve the questioner’s problem. The native-language answers drive the effect, which is larger when the question quality is higher and the incentives are stronger. Second, users who participate in their native language but not in English contribute lower-quality answers compared with those who contribute in English. This suggests that lower language barriers allow users with less expertise in the subject to participate. These results show that the platform should adopt multiple languages to maximise the quality of the information collected, although doing so may also result in an inflow of low-quality content from new users. This paper was accepted by Joshua Gans, business strategy. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.01333 .

Deciphering digital popularity from question quality: the preeminence of social attributes over textual dimensions in driving CQA platform traffic

ABSTRACT Community Question Answering (CQA) sites, now navigating through millions of daily user visits, hinge significantly on the quality of user-generated questions to bolster user satisfaction and, consequently, website traffic – crucial for platform sustainability. While the quality of questions ostensibly sets the stage for insightful answers, thereby attracting substantial traffic, existing research has predominantly sidelined the significance of questions, favoring a focus on traditional factors, such as answer quality, to elucidate traffic determinants. In this study we quantified the interrelation between question quality and traffic by extracting 15 relevant question-quality features and corresponding traffic (i.e. pageview) data from a leading Chinese CQA platform. Utilizing Grey Relational Analysis, we assessed these attributes’ relative influence on pageviews and created a ranking list of features of questions. Based on the rankings, our findings spotlight the paramount influence of a question's digital popularity and attributes tied to the questioner – central pillars of social features – surpassing the impact of textual elements. This study further delineates the underlying reasons for such patterns based on traffic sourced from diverse channels. Our revelations offer pivotal insights for optimizing CQA platform engagement and furnish avenues for prospective research.

Anxiety and disease awareness in individuals with heredity for abdominal aortic aneurysm.

The psychological consequences of being aware of an increased risk of developing abdominal aortic aneurysm as a first-degree relative of a person with abdominal aortic aneurysm are hitherto unexplored. This study investigates the awareness of heritability and anxiety in male and female adult offspring of abdominal aortic aneurysm patients compared to controls. Health-related quality of life among participants with aortic pathology was compared to participants with normal aortic diameters. This was a cross-sectional point prevalence study based on the participants examined in the Detecting Abdominal Aortic Aneurysm in First Degree Relatives Trial (DAAAD; 752 adult offspring, 756 matched controls), 2020-2022. Questionnaires about health-related quality of life and study-specific questions regarding awareness of heritability were collected prior to the aortic ultrasound. Attendance rate was higher among individuals with heredity compared to controls (67% vs. 52%, p < 0.001). Of 1508 adult offspring examined, 65% reported having a close relative with abdominal aortic aneurysm (6% in controls). Female adult offspring reported higher awareness of heritability than controls (38% vs. 12%, p < 0.001), as did males (32% vs. 8%, p < 0.001). A slight majority of participants with awareness reported anxiety (54% of female offspring; 51% of male). There were no measured differences in health-related quality of life between the groups when standard health-related quality of life instruments were used. The higher-than-expected proportion of adult offspring with awareness of heritability and anxiety about such risk indicates that we fail to communicate risk to this group appropriately via the current channels of information within the healthcare system. This calls for the development of dedicated strategies for improved communication of abdominal aortic aneurysm risk to patients and their next of kin.

Contaminant of Emerging Concerns in Modder River Catchment of Free State: Implication for Environmental Risk and Water Sources Protection

This study was aimed at monitoring the occurrence and potential sources of emerging contaminants in water sources within the Modder River catchment. Selected water quality indicators were analysed by Hanna multi-parameter meters. Emerging contaminants such as acetaminophen, carbamazepine, ibuprofen, atrazine, simazine, metolachlor, terbuthylazine, 17-alpha-ethinyl-estradiol, estradiol, progesterone, and testosterone were analysed by high performance liquid chromatography mass spectrometry. The sources of emerging contaminants were determined by statistical methods such as Pearson correlation and hierarchical cluster analysis. Results showed that all the sampled water sources have some level of questionable drinking water quality and necessitate some amount of treatment to reduce the contamination before consumption, especially DO, EC, and pH. The 17-alpha-ethinyl-estradiol mean values in rivers (7.79 and 31.55 µg/L), dams (1.83 and 6.90 µg/L), and treated drinking water (0.2 and 0.73 µg/L) were the highest in summer and autumn seasons, respectively. Wastewater effluents, domestic sewage, urban surface runoff, agricultural runoff, and illegal dumping were identified as the possible sources of emerging contaminants pollution. Waste management education, proper application of herbicides, and advance wastewater treatment methods were some of the suggested mitigation strategies. The outcomes may be relevant for environmental protection and water sustainability in the catchment.

Patients Experience Improvements in Pain and Quality of Life When Treated at a Physician Led Integrative Medicine Clinic at a Large Air Force Military Treatment Facility

ObjectiveA large percentage of military treatment facilities (MTFs) offer at least one type of complementary medicine and 80% of patients surveyed felt that integrative therapies should be routinely offered at all MTFs. We are the first to offer the breadth of modalities outlined below in a dedicated MTF Integrative Medicine Clinic (IMC). The purpose of this study was to determine whether the basket of services offered at our IMC improved patient pain and quality of life. Design and SettingPatient outcomes over a three-year period were retrospectively evaluated from a single cohort. Patients referred to the IMC for chronic pain at the MTF recorded pain scores using the Defense and Veterans Pain Rating Scale (DVPRS) and completed the Dallas Pain Questionnaire (DPQ), a 16-item visual analog quality of life questionnaire, at each visit. They also completed a non-validated quality of life question on discharge from the clinic. Modalities used include microcurrent therapies, extracorporeal shockwave therapy, medical Acupuncture, myofascial release techniques, low-level laser therapy, and the AllCore360. Results761 patients were seen from August 2019 to October 2022 in over 3,280 clinic visits. On average per visit, DVPRS scores were reduced from 4.4 to 1.5 (MD 2.9; 95%CI, 2.7-3.1). Total DPQ scores were reduced by 16.5% (MD 27.7; 95%CI 22.7-32.7). On discharge from the clinic 81% of patients reported their quality of life was either “a lot better” or “somewhat better.” ConclusionsIf implemented, we believe comparable clinics across the Department of Defense can replicate these improvements in patient-oriented outcomes.