Automated Assessment System Research Articles

Automatic assessment of divergent thinking in Chinese language with TransDis: A transformer-based language model approach.

Language models have been increasingly popular for automatic creativity assessment, generating semantic distances to objectively measure the quality of creative ideas. However, there is currently a lack of an automatic assessment system for evaluating creative ideas in the Chinese language. To address this gap, we developed TransDis, a scoring system using transformer-based language models, capable of providing valid originality (novelty) and flexibility (variety) scores for Alternative Uses Task (AUT) responses in Chinese. Study 1 demonstrated that the latent model-rated originality factor, comprised of three transformer-based models, strongly predicted human originality ratings, and the model-rated flexibility strongly correlated with human flexibility ratings as well. Criterion validity analyses indicated that model-rated originality and flexibility positively correlated to other creativity measures, demonstrating similar validity to human ratings. Study 2 and 3 showed that TransDis effectively distinguished participants instructed to provide creative vs. common uses (Study 2) and participants instructed to generate ideas in a flexible vs. persistent way (Study 3). Our findings suggest that TransDis can be a reliable and low-cost tool for measuring idea originality and flexibility in Chinese language, potentially paving the way for automatic creativity assessment in other languages. We offer an open platform to compute originality and flexibility for AUT responses in Chinese and over 50 other languages ( https://osf.io/59jv2/ ).

Design and implementation of an automatic nursing assessment system based on CDSS technology

BackgroundVarious quantitative and quality assessment tools are currently used in nursing to evaluate a patient’s physiological, psychological, and socioeconomic status. The results play important roles in evaluating the efficiency of healthcare, improving the treatment plans, and lowing relevant clinical risks. However, the manual process of the assessment imposes a substantial burden and can lead to errors in digitalization. To fill these gaps, we proposed an automatic nursing assessment system based on clinical decision support system (CDSS). The framework underlying the CDSS included experts, evaluation criteria, and voting roles for selecting electronic assessment sheets over paper ones. MethodsWe developed the framework based on an expert voting flow to choose electronic assessment sheets. The CDSS was constructed based on a nursing process workflow model. A multilayer architecture with independent modules was used. The performance of the proposed system was evaluated by comparing the adverse events’ incidence and the average time for regular daily assessment before and after the implementation. ResultsAfter implementation of the system, the adverse nursing events’ incidence decreased significantly from 0.43 % to 0.37 % in the first year and further to 0.27 % in the second year (p-value: 0.04). Meanwhile, the median time for regular daily assessments further decreased from 63 s to 51 s. ConclusionsThe automatic assessment system helps to reduce nurses’ workload and the incidence of adverse nursing events.

Automatic Short Answer Grading System Based on Semantic Networks and Support Vector Machine

In education, exams are used to asses students’ acquired knowledge; however, the manual assessment of exams consumes a lot of teachers’ time and effort. In addition, educational institutions recently leaned toward distance education and e-learning due the Coronavirus pandemic. Thus, they needed to conduct exams electronically, which requires an automated assessment system. Although it is easy to develop an automated assessment system for objective questions. However, subjective questions require answers comprised of free text and are harder to automatically assess since grading them needs to semantically compare the students’ answers with the correct ones. In this paper, we present an automatic short answer grading method by measuring the semantic similarity between the students answer and the correct answer. A semantic network was built to represent the relationship between the words of the two texts to calculate semantic similarity. Representing a text as a semantic network is the best knowledge representation that comes closest to human comprehension of the text, where the semantic network reflects the sentence's semantic, syntactical, and structural knowledge. Several features were extracted from the semantic network and used as input to train the support vector machine (SVM) model to predict the degree of the targeted semantic similarity. The proposed method was tested on the Mohler dataset that is publicly available online. The obtained results were evaluated and reported in terms of Pearson correlation and root mean squared error (RMSE) where it achieved 0.63 and 0.83 respectively. The proposed method outperformed all previous methods on the used dataset.

Teachers in the loop? An analysis of automatic assessment systems under Article 22 GDPR

Teachers in the loop? An analysis of automatic assessment systems under Article 22 GDPR

Penerapan Artificial Intelligence Menggunakan Fuzzy Logic dalam Dunia Pendidikan

This research aims to apply artificial intelligence or what is commonly known as Artificial Intelligence (AI) using fuzzy logic in the world of education. The topic of artificial intelligence has become a hot and important topic of conversation today. Fuzzy logic allows processing unclear information and is used to overcome uncertainty that exists in the world of education. This article discusses the use of fuzzy logic techniques to improve the quality of education. One application of fuzzy logic is in the development of an automatic assessment system used by lecturers in providing assessments to students in the learning process. Fuzzy logic can carry out assessments objectively and enable more appropriate curriculum adjustments based on students' interpersonal skills which can ultimately increase the efficiency of the learning process. The use of fuzzy logic can provide recommendations for the learning process that can help students find material that suits their interests and needs. Fuzzy logic can be used to measure student development in social and emotional cognitive aspects. The application of AI with fuzzy logic aims to improve the quality of education and student learning experiences in the world of education.

DLMDish: Using Applied Deep Learning and Computer Vision to Automatically Classify Mauritian Dishes

The benefits of using an automatic dietary assessment system for accompanying diabetes patients and prediabetic persons to control the risk factor also referred to as the obesity “pandemic” are now widely proven and accepted. However, there is no universal solution as eating habits of people are dependent on context and culture. This project is the cornerstone for future works of researchers and health professionals in the field of automatic dietary assessment of Mauritian dishes. We propose a process to produce a food dataset for Mauritian dishes using the Generative Adversarial Network (GAN) and a fine Convolutional Neural Network (CNN) model for identifying Mauritian food dishes. The outputs and findings of this research can be used in the process of automatic calorie calculation and food recommendation, primarily using ubiquitous devices like mobile phones via mobile applications. Using the Adam optimizer with carefully fixed hyper-parameters, we achieved an Accuracy of 95.66% and Loss of 3.5% as concerns the recognition task.

Development of an automatic evaluation system for photooxidation and assessment of polyethylenes containing HALS and UVA

An automatic evaluation system for photooxidation has been developed by utilizing a chemiluminescence apparatus combined with a light irradiation system. The system is composed of a controller, a driving device, and irradiation equipment, which are integrated with the chemiluminescence measurement system. Various measurement conditions can be set independently, including irradiation time (fixed or variable), waiting time from irradiation to measurement, chemiluminescence measurement time, number of repetitions (up to 999 times), and measurement temperature. LLDPEs containing Hindered Amine Light Stabilizers (HALS) and Ultraviolet Absorbers (UVA) were evaluated using this system. The decay of chemiluminescence after UV irradiation was repeatedly measured at r.t. and 105℃, respectively, below the melting point of LLDPE, and the time-course of chemiluminescence was obtained by plotting time against I(t = 0) for each instance. I(t = 0) represent the rate of photooxidation under UV exposure, calculated using second-order analysis of the chemiluminescence decay. The chemiluminescence intensity increased significantly in the initial stage with irradiation time, maintained a stable intensity for a period, and then exhibited a substantial increase in the later stage. We termed the point at which this significant increase in chemiluminescence occurs as the induction time of photooxidation (photo-OIT). Using this system, the photo-OIT of LLDPE samples with HALS and UVA was measured, revealing the potential for assessing the synergistic effects of photooxidation stabilizers.

A python system for regional landslide susceptibility assessment by integrating machine learning models and its application

Landslide susceptibility assessment is considered the first step in landslide risk assessment, but current studies mostly rely on GIS platforms or other software for data preprocessing. The modeling process is relatively complicated and multi-models cannot be integrated. With regard to this issue, this study develops a Python system for automatic assessment of regional landslide susceptibility. The Python system implements landslide susceptibility assessment through three modules: geographic data processing, machine learning modeling and result evaluation analysis. For geographic data processing, ten landslide influencing factors can be used to construct an evaluation factor dataset and reclassify the thematic maps based on the frequency ratio method. Four built-in machine learning models (logistic regression (LR), multi-layer perceptron (MLP), support vector machine (SVM) and extreme gradient boosting (XGBoost)) are integrated into the system to complete susceptibility modeling and calculation. Additionally, receiver operating characteristic (ROC) curves can be automatically generated to evaluate the accuracy. The system was then applied into Lantian County in Shaanxi Province as a demonstration example. The results show that the areas under the ROC curve (AUC) of the four models are 0.838 (LR)、0.882 (SVM)、0.809 (MLP) and 0.812 (XGBoost), respectively, indicating that the SVM model was the most suitable model for landslide susceptibility assessment in Lantian County in the Loess Plateau of China. The system has now been made open source on Github, which can effectively improve the efficiency of regional landslide susceptibility assessment, especially provide tools for data processing and modeling for non-professionals.

Automated Assessment System Using Machine Learning Libraries

Assessment and the grading of students is a task that has been done for as long as school has existed. This was previously done by teachers in primary and secondary, lecturers for institutions like JAMB and lecturers in schools. Up until now students’ marks were influenced by other external factors such as bad handwriting, lengthy paragraphs, roundabout way of speaking rather than going straight to the point and the sheer number of assignments the lecturer has to mark. This has resulted in students getting lower or higher marks than they should be awarded. This project is to create an ML (Machine Learning) powered assessment system that will take the assignment questions and the marking scheme and award the student the marks similar to what the ideal lecturer would have given. This will also reduce the time the lecturers spend on marking and ensure the students get their results on time. This project will be made with Python and machine learning and will be tested with a number of potential answers to the questions and their grading’s. This will enable system to be able to grade assignments as soon as they are uploaded. This research will be limited by the fact that the system can only handle the marking of short sentences accurately and not long paragraphs. The system is also limited by the fact that it can only mark with the aid of the marking scheme and not without it so it is not a truly intelligent model in that regard. The research showed that the system is indeed capable of obtaining the similarity between two paragraphed answers provided but it needs extras to produce the most accurate results.

An automatic risk assessment system for sudden cardiac death using look ahead pattern

An automatic risk assessment system for sudden cardiac death using look ahead pattern

Prediction of the hand function part of the Fugl‐Meyer scale after stroke using an automatic quantitative assessment system

AbstractHand function assessment is an essential component of the process of stroke rehabilitation because of the high incidence of hand motor dysfunction. In terms of the manual evaluation of hand function, the Fugl‐Meyer scale is a recommended scale with high reliability and validity. However, the need for accurate assessments and increasing developments in technology has led to the promotion of automatic quantitative assessment systems for the hand. In this study, we collected quantitative data on hand function with an automatic system and the upper limb Fugl‐Meyer assessment (FMA) from 79 people with stroke. We developed decision tree (DT) and gradient‐boosted decision tree (GBDT) predictive models for the Fugl‐Meyer score using features extracted from the Hand Automatic Quantitative Assessment System (HAQAS). Predictive performances were compared between these models regarding the predictive accuracy and Cohen's kappa. There were high correlations between features automatically collected by the HAQAS and the Fugl‐Meyer scale in all the sub‐items, with the maximal correlations all being over 0.5, indicating the high validity of the HAQAS in automatic FMA prediction. Hand functions were more highly correlated (average correlation coefficient 0.90) with HAQAS features than wrist functions (average correlation coefficient 0.54), and the GBDT achieved higher predictive accuracies and agreement than the DT algorithm. We conclude that the HAQAS is feasible for stroke patients with hand dysfunction and convenient for clinicians and therapists. This study was registered in the Chinese Clinical Trial Registry (ChiCTR1800019098).

Automated assessment system for programming courses: a case study for teaching data structures and algorithms

AbstractAn important course in the computer science discipline is ‘Data Structures and Algorithms’ (DSA). The coursework lays emphasis on experiential learning for building students’ programming and algorithmic reasoning abilities. Teachers set up a repertoire of formative programming exercises to engage students with different programmatic scenarios to build their know-what, know-how and know-why competencies. Automated assessment tools can assist teachers in inspecting, marking, and grading of programming exercises and also support them in providing students with formative feedback in real-time. This article describes the design of a bespoke automarker that was integrated into the DSA coursework and therefore served as an instructional tool. Activity theory has provided the pedagogical lens to examine how the automarker-mediated instructional strategy enabled self-reflection and assisted students in their formative learning journey. Learner experiences gathered from 39 students enrolled in DSA course shows that the automarker facilitated practice-based learning to advance students know-what, know-why and know-how skills. This study contributes to both curricula and pedagogic practice by showcasing the integration of an automated assessment strategy with programming-related coursework to inform future teaching and assessment practice.

USER EXPERIENCE ANSWER SYSTEM AUTOMATICALLY WITH USER CENTERED DESIGN AND USER EXPERIENCE QUESTIONNAIRE-SHORT

The interface of the automatic answer assessment system is plagued by several issues, including an unfamiliar layout, unresponsive design, inconsistency in elements, and a lack of clarity in presenting information. These problems significantly hinder the overall user experience. As a result, this study aimed to address these challenges by designing a user-centered experience for the automatic answer assessment system, using a high-fidelity prototype tailored to meet user needs. The user-centered design method involved four crucial stages: Specify The Context Of Use, Specify Requirements, Create Designs, and Evaluate Designs. Through rigorous usability testing with teachers, the design achieved an impressive effectiveness rating of 90%, firmly establishing it as a "very effective" solution. Additionally, it demonstrated high efficiency with a value of 0.01307 goals/sec, and teachers expressed positive feedback, confirming the satisfaction and usability of the new interface. Similarly, students' usability testing yielded noteworthy results, with a 90% effectiveness rating, also classified as "very effective." The interface showcased a high level of efficiency, with a value of 0.0849 goals/sec. While the satisfaction value fell below the PSSUQ norm, students still found the interface to be user-friendly and satisfactory. Furthermore, the user experience testing, utilizing the UEQ-S, provided valuable insights. For teachers, the pragmatic aspect scored 1.85, the hedonic aspect scored 2.33, and the overall aspect received a commendable score of 2.09, all of which fell within the excellent category on benchmarks. Similarly, students' ratings were highly positive, with scores of 2.14 for both pragmatic and hedonic aspects, and an overall score of 2.14, signifying an excellent user experience.The retrospective think-aloud validation test reaffirmed the positive response from prospective users. Overall, this research, employing a user-centered design approach, successfully delivered a highly satisfactory and effective user experience for both teachers and students using the automatic answer assessment system.

An Automatic Remote Health Risk Assessment system based on LSTM for elderly.

To address the challenges posed by the aging process, we designed and validated an LSTM-based automatic remote health risk assessment system for the elderly. This system consists of a wireless physiological parameter sensing unit, a vital sign prediction unit and a pre-defined risk scoring criteria unit. The vital sign prediction module is composed of five 5-input-1-output neural networks based on the LSTM architecture, which are responsible for predicting the vital signs collected by wireless sensors, including: systolic blood pressure (SBP), pulse rate (PR), respiratory rate (RR), temperature (TEMP), and oxygen saturation (SPO2). The pre-defined health risk scoring criteria is a simplified version of the National Early Warning Score (NEWS), which is responsible for calculating the risk level based on the predicted values. This allows the care team to respond to the medical needs of the elderly in a timely manner. Through experiments, our system can achieve a risk identification accuracy of 74% and MAEs of the predicted values for each parameter are in an acceptable range. Our results suggest that an automated remote health risk assessment system for the elderly using deep learning could be a viable new strategy for home-based monitoring systems.

Design and Implementation of a Programming Automatic Assessment System in Jupyter Notebook

Learning programming is challenging. So, computer educators have developed various tools to help students. In this paper, we have developed a tool that combines the advantages of a Programming Automatic Assessment (PAA) system and Jupyter Notebook (JN) to support learning programming. The design direction of this system is free to use, easy to set up, and supports interactive computing. The Programming Automatic Assessment in Jupyter Notebook (PAAinJN) is available free of charge using the assessment module released on Git and the personal JN. The initialization is completed by executing in a code cell with two lines of code that downloads and executes the assessment module. In an interactive computing environment, presenting problems, writing code to be evaluated, and evaluating code can be executed in the code cells, and the problems and the results of the assessment are presented in the code cell outputs. The performance was verified by the examples presented in a high school informatics textbook using the programming automatic assessment system as teaching learning material. In addition, we propose a way to develop teaching-learning materials using PAAinJN in consideration of teachers and students and a way of distributing and collecting teaching-learning materials using the free Learning Management System. PAAinJN is expected to help students learn programming by eliminating assessment and feedback delays through PAA while learning to program in an interactive computing environment.

Evaluation of Submission Limits and Regression Penalties to Improve Student Behavior with Automatic Assessment Systems

Objectives . Automatic assessment systems are widely used to provide rapid feedback for students and reduce grading time. Despite the benefits of increased efficiency and improved pedagogical outcomes, an ongoing challenge is mitigating poor student behaviors when interacting with automatic assessment systems including numerous submissions, trial-and-error, and relying on marking feedback for problem solving. These behaviors negatively affect student learning as well as have significant impact on system resources. This research quantitatively examines how utilizing submission policies such as limiting the number of submissions and applying regression penalties can reduce negative student behaviors. The hypothesis is that both submission policies will have a significant impact on student behavior and reduce both the number of submissions and regressions in student performance. The research questions evaluate the impact on student behavior, determine which submission policy is the most effective, and what submission policy is preferred by students. Participants . The study involved two course sections in two different semesters consisting of a total of 224 students at the University of British Columbia, a research-intensive university. The students were evaluated using an automated assessment system in a large third year database course. Study Methods . The two course sections used an automated assessment system for constructing database design diagrams for assignments and exams. The first section had no limits on the number of submissions for both assignments and exams. The second section had limits for the exams but no limits on assignments. On the midterm, participants were randomly assigned to have either a restriction on the total number of submissions or unlimited submissions but with regression penalties if a graded answer was lower than a previous submission. On the final exam, students were given the option of selecting their submission policy. Student academic performance and submission profiles were compared between the course sections and the different submission policies. Findings. Unrestricted use of automatic grading systems results in high occurrence of undesirable student behavior including trial-and-error guessing and reduced time between submissions without sufficient independent thought. Both submission policies of limiting maximum submissions and utilizing regression penalties significantly reduce these behaviors by up to 85%. Overall, students prefer maximum submission limits, and demonstrate improved behavior and educational outcomes. Conclusions . Automated assessment systems when used for larger problems related to design and programming have benefits when deployed with submission restrictions (maximum attempts or regression penalty) for both improved student learning behaviors and to reduce the computational costs for the system. This is especially important for summative assessment but reasonable limits for formative assessments are also valuable.

Automatic Surgical Skill Assessment System Based on Concordance of Standardized Surgical Field Development Using Artificial Intelligence

Automatic surgical skill assessment with artificial intelligence (AI) is more objective than manual video review-based skill assessment and can reduce human burden. Standardization of surgical field development is an important aspect of this skill assessment. To develop a deep learning model that can recognize the standardized surgical fields in laparoscopic sigmoid colon resection and to evaluate the feasibility of automatic surgical skill assessment based on the concordance of the standardized surgical field development using the proposed deep learning model. This retrospective diagnostic study used intraoperative videos of laparoscopic colorectal surgery submitted to the Japan Society for Endoscopic Surgery between August 2016 and November 2017. Data were analyzed from April 2020 to September 2022. Videos of surgery performed by expert surgeons with Endoscopic Surgical Skill Qualification System (ESSQS) scores higher than 75 were used to construct a deep learning model able to recognize a standardized surgical field and output its similarity to standardized surgical field development as an AI confidence score (AICS). Other videos were extracted as the validation set. Videos with scores less than or greater than 2 SDs from the mean were defined as the low- and high-score groups, respectively. The correlation between AICS and ESSQS score and the screening performance using AICS for low- and high-score groups were analyzed. The sample included 650 intraoperative videos, 60 of which were used for model construction and 60 for validation. The Spearman rank correlation coefficient between the AICS and ESSQS score was 0.81. The receiver operating characteristic (ROC) curves for the screening of the low- and high-score groups were plotted, and the areas under the ROC curve for the low- and high-score group screening were 0.93 and 0.94, respectively. The AICS from the developed model strongly correlated with the ESSQS score, demonstrating the model's feasibility for use as a method of automatic surgical skill assessment. The findings also suggest the feasibility of the proposed model for creating an automated screening system for surgical skills and its potential application to other types of endoscopic procedures.

A CT-based deep learning system for automatic assessment of aortic root morphology for TAVI planning

Accurate planning of transcatheter aortic valve implantation (TAVI) is important to minimize complications, and it requires anatomic evaluation of the aortic root (AR), commonly performed through 3D computed tomography (CT) image analysis. Currently, there is no standard automated solution for this process.Two convolutional neural networks with 3D U-Net architectures (model 1 and model 2) were trained on 310 CT scans for AR analysis. Model 1 performs AR segmentation and model 2 identifies the aortic annulus and sinotubular junction (STJ) contours. After training, the two models were integrated into a fully automated pipeline for geometric analysis of the AR. Results were validated against manual measurements of 178 TAVI candidates.The trained CNNs segmented the AR, annulus, and STJ effectively, resulting in mean Dice scores of 0.93 for the AR, and mean surface distances of 0.73 mm and 0.99 mm for the annulus and STJ, respectively. Automatic measurements were in good agreement with manual annotations, yielding annulus diameters that differed by 0.52 [-2.96, 4.00] mm (bias and 95% limits of agreement for manual minus algorithm). Evaluating the area-derived diameter, bias, and limits of agreement were 0.07 [-0.25, 0.39] mm. STJ and sinuses diameters computed by the automatic method yielded differences of 0.16 [-2.03, 2.34] and 0.1 [-2.93, 3.13] mm, respectively.The proposed tool is a fully automatic solution to quantify morphological biomarkers for pre-TAVI planning. The method was validated against manual annotation from clinical experts and showed to be quick and effective in assessing AR anatomy, with potential for time and cost savings.

Automatic Assessment of Stereotactic Radiation Therapy Outcome in Brain Metastasis Using Longitudinal Segmentation on Serial MRI.

The standard clinical approach to assess the radiotherapy outcome in brain metastasis is through monitoring the changes in tumour size on longitudinal MRI. This assessment requires contouring the tumour on many volumetric images acquired before and at several follow-up scans after the treatment that is routinely done manually by oncologists with a substantial burden on the clinical workflow. In this work, we introduce a novel system for automatic assessment of stereotactic radiation therapy (SRT) outcome in brain metastasis using standard serial MRI. At the heart of the proposed system is a deep learning-based segmentation framework to delineate tumours longitudinally on serial MRI with high precision. Longitudinal changes in tumour size are then analyzed automatically to assess the local response and detect possible adverse radiation effects (ARE) after SRT. The system was trained and optimized using the data acquired from 96 patients (130 tumours) and evaluated on an independent test set of 20 patients (22 tumours; 95 MRI scans). The comparison between automatic therapy outcome evaluation and manual assessments by expert oncologists demonstrates a good agreement with an accuracy, sensitivity, and specificity of 91%, 89%, and 92%, respectively, in detecting local control/failure and 91%, 100%, and 89% in detecting ARE on the independent test set. This study is a step forward towards automatic monitoring and evaluation of radiotherapy outcome in brain tumours that can streamline the radio-oncology workflow substantially.

Hybrid Approach for Skin Disease Classification: Integrating Machine learning and Deep Learning

Abstract: Skin infections are more common than many other diseases, and they can be caused by various factors such as infectious pollution, microorganisms, awareness, and infections. With the advancement of lasers and photonics-based medical technology, the diagnosis of skin infections has become faster and more accurate. However, the cost of such diagnosis is still limited and expensive. Therefore, image processing techniques are used to develop an automated assessment system for dermatology at an early stage. The extraction of features plays a crucial role in accurately and quickly diagnosing skin diseases. PC vision plays an important role in the detection of skin diseases in various ways. This study focuses on four skin diseases: ringworm, nail parasite, psoriasis, and atopic dermatitis. Convolutional neural networks have achieved close to or even better performance than humans in the imaging field. The skin diseases are classified using a machine learning algorithm, i.e., random forest, which achieves an accuracy of 98.23% after 100 epochs.