Automated Scoring System Research Articles

Automatic English Pronunciation Evaluation Algorithm Based on Sequence Matching and Feature Fusion

This article focuses on the question answering type of automatic scoring system for large-scale spoken English examinations and scores using a method called multifeature fusion. Three types of features are extracted to score using speech recognition text as the research object. The three types of features are similarity, syntactic, and phonetic. There are nine distinct characteristics that describe the relationship between examinee responses and expert ratings. Manhattan distance is improved as a measure of similarity in the similarity feature. Simultaneously, a feature of keyword coverage based on editing distance is proposed, and the phenomenon of word variation in text recognition is fully considered, in order to provide examinees with an objective and fair score. To obtain the machine score, all extracted features were fused using a multiple linear regression model. The experimental results demonstrate that the extracted features are extremely effective for machine scoring, with the system scoring performance of the examinee as a unit equaling 98.4 percent of expert scoring performance.

Automatic Scoring of English Essays Based on Machine Learning Technology in a Wireless Network Environment

The English composition is an important indicator of English learners’ overall language skills and is asked in large-scale English examinations, both in China’s college entrance examinations and graduate examinations and in the TOEFL, GRE, and IELTS examinations in Europe and the United States. Some automatic scoring systems for English writing have been created in the United States and internationally, however the systems still have issues with generalization, accuracy, and error correction. In this paper, we present a method to improve the accuracy of existing automatic composition scoring systems through deep learning techniques in a wireless network environment. Experiments reveal that the method can accurately assess the quality of English learners’ writings, paving the way for the creation of an automated composition scoring system for large-scale machine testing and web-based self-learning platforms.

0606 A Paradigm for Testing the Accuracy of Digital Sleep Staging Systems

Abstract Introduction Despite evidence showing that agreement between human and some automatic staging systems is generally comparable to agreement between human scorers, automated scoring is rarely used in clinical practice, even though it offers time savings and consistency. We propose a paradigm for testing digital systems that reveals their true accuracy vs. highly experienced academic scorers. As an example of a digital method to be tested, we used Michele Sleep Scoring (abbreviated:Digital). Methods 70 PSGs were scored by 6 experienced technologists from 3 academic centers. Staging results were compared to digital staging results using an epoch-by-epoch approach. For each PSG we carried out 6 cycles of comparisons. Each cycle consisted of two steps, one comparing one scorer (tested scorer) with the scoring of the five remaining scorers (judges), and one comparing Digital as the tested scorer with the same 5 judges. Error 1 was assessed when all judges disagreed with the tested scorer but there was disagreement between the judges. Error 2 was assigned when all judges disagreed with the tested scorer but agreed unanimously on the stage. For each PSG the number of epochs with types 1 and 2 errors was counted for each scorer (n=6 scorers) and for Digital. Results of all 70 PSGs were pooled, and percent of types 1 and 2 errors is reported for all scorers and Digital. Results 70 PSGs (females aged 51.1 ± 4.2 years) were evaluated. Average times in different sleep stages (manual scoring) were 43±18, 244±47, 30±21, and 81±25 minutes for stages N1, N2, N3 and REM, respectively. TST was 398±52 minutes, and sleep efficiency was 84±8%. There was a total of 65,053 epochs scored by each scorer and Digital. The average percent of type 1 errors made by scorers for all epochs was 6.4% (0-33.2) vs. 7.8% (1.68-26.6) made by Digital. The average percent of type 2 errors made by scorers for all epochs was 3.9% (0-28.6) vs. 4.3% (0-17.3) made by Digital. Conclusion This study provides an objective way of testing the accuracy of automated scoring systems and supports evidence that the accuracy of Michele Sleep Scoring is comparable to manual scoring. Support (If Any) None

Automated Progress-Monitoring for Literate Language Use in Narrative Assessment (LLUNA).

Language sample analysis (LSA) is an important practice for providing a culturally sensitive and accurate assessment of a child's language abilities. A child's usage of literate language devices in narrative samples has been shown to be a critical target for evaluation. While automated scoring systems have begun to appear in the field, no such system exists for conducting progress-monitoring on literate language usage within narratives. The current study aimed to develop a hard-coded scoring system called the Literate Language Use in Narrative Assessment (LLUNA), to automatically evaluate six aspects of literate language in non-coded narrative transcripts. LLUNA was designed to individually score six literate language elements (e.g., coordinating and subordinating conjunctions, meta-linguistic and meta-cognitive verbs, adverbs, and elaborated noun phrases). The interrater reliability of LLUNA with an expert scorer, as well as its' reliability compared to certified undergraduate scorers was calculated using a quadratic weighted kappa (Kqw). Results indicated that LLUNA met strong levels of interrater reliability with an expert scorer on all six elements. LLUNA also surpassed the reliability levels of certified, but non-expert scorers on four of the six elements and came close to matching reliability levels on the remaining two. LLUNA shows promise as means for automating the scoring of literate language in LSA and narrative samples for the purpose of assessment and progress-monitoring.

Accuracy of key point matrix technology based contactless automatic measurement for joint motion of hand

To validate the use of key point matrix technology based contactless automatic measurement for evaluation of joint motion of hand. Thirty-three volunteers were enrolled to evaluate the extension and flexion of hand joints between May 2021 and November 2021. There were 20 males and 13 females, the age ranged from 16 to 70 years with an average of 30.2 years. The extension angles of 14 joints of 5 fingers (including hyperextension) and the flexion angles of 12 joints of 4 fingers (excluding thumb) of volunteers were measured by key point matrix technology and manual goniometer, respectively. Then 5 participants and repeated measurement experiment were employed to test the system repeatability and accuracy; 28 participants and paired measurement experiment were employed to test the system accuracy. The average repeatability of finger joint motion measured by the key point matrix technology was 1.801° (extension) and 7.823° (flexion), respectively. Compared with manual measurement, the average differences of each finger joint measured by the key point matrix technology were 3.225° in extension and 14.145° in flexion, respectively. The key point matrix technology based contactless automatic evaluation system offered excellent consistency with the manual goniometers ( ICC=0.875). While most of the consistency with manual goniometer of individual joints were at moderate levels (median of ICC, 0.440). The correlation coefficients between the measurement results of the two methods were mainly positive in the extension of the joint ( P<0.05) and negative in the flexion of the joints ( P<0.05). The key point matrix technology based contactless automatic evaluation provides sufficient measurement repeatability and accuracy in evaluation for the joint motion of hand.

Functionalities, Benchmarking System and Performance Evaluation for a Domestic Service Robot: People Perception, People Following, and Pick and Placing

This paper describes the development of three main functionalities for a domestic mobile service robot and an automatic benchmarking system used for the systematic performance evaluation of the robot’s functionalities. Three main robot functionalities are addressed: (1) People Perception, (2) People Following and (3) Pick and Placing, where the hardware and software systems developed for each functionality are described and demonstrated on an actual mobile service robot, with the goal of providing assistance to an elderly person inside the house. Furthermore, a set of innovative benchmarks and an automatic performance evaluation system are proposed and used to evaluate the performance of the developed functionalities. These benchmarks are now made publicly available and is part of the European Robotics League (ERL)-Consumer to systematically evaluate the performance of service robot solutions at different testbeds around Europe.

Short text automatic scoring system based on BERT-BiLSTM model

Aiming at the problems of sparse features, polysemy of one word and less context related information in short text automatic scoring, a short text automatic scoring model based on bidirectional encoder representations from transformers - bidirectional long short-term memory (BERT-BiLSTM) is proposed. Firstly, the large-scale corpus is pre-trained with bidirectional encoder representations from transformers (BERT) language model to acquire the semantic features of the general language. Then the semantic features of short text and the semantics of keywords in a specific context are acquired through the short text data for the pre-fine tuning downstream specific tasks set pre-fined by BERT. And then the deep-seated context dependency is captured through bidirectional long short-term memory (BiLSTM). Finally, the obtained feature vectors are input into Softmax regression model for automatic scoring. The experimental results show that compared with other benchmark models of convolutional neural networks(CNN), character-level CNN (CharCNN), long short-term memory (LSTM) and BERT, the short text automatic scoring model based on BERT-BiLSTM achieves the best average value of quadratic weighted kappa coefficient.

Football Match Scoring Method Based on Adaptive Neural Network Algorithm

To study in more detail the impact of various indicators on the scoring system of a football match, the author suggests a football game scoring method based on an adaptive neural network algorithm. Firstly, the application background of football match prediction, the research and application status of the adaptive neural network algorithm, and the related research of football match prediction are described; Secondly, the factors affecting the outcome of football games are analyzed, and the applicability of the adaptive neural network algorithm in football match prediction are summarized. Through the system requirement analysis, the overall design of the system and the design of the database are completed. From here, we see that there is a huge opportunity to use an adaptive neural network in an automatic scoring system after a football match, which can replace manual scoring and reduce the manual workload.

The higher the score, the better the clinical outcome: retrospective evaluation of automatic embryo grading as a support tool for embryo selection in IVF laboratories.

Is the automatic embryo grading function of specific time-lapse systems clinically useful as a decision support tool for IVF laboratories? Blastocyst grading according to the automatic scoring system is directly associated with the likelihood of implantation and live birth, at least in treatments without preimplantation genetic testing for aneuploidy (PGT-A). Several embryo selection algorithms have been described since the introduction of time-lapse technology in IVF laboratories, but no one algorithm has yet been sufficiently consolidated for universal use. Multicentric models based on automated grading systems offer promise for standardization of embryo selection. A retrospective cohort study was performed including 1678 patients who underwent IVF treatments between 2018 and 2020 and whose embryos (n = 12468) were cultured in time-lapse systems. After obtaining the required parameters (division time to 2, 3, 4 and 5 cells; time of blastocyst formation; inner cell mass quality; and trophectoderm quality), the automatic embryo score was calculated using the software included in the appropriate workstation. First, embryo score was compared with conventional morphological quality and the subsequent clinical outcomes of 1952 single blastocyst transfers. Second, we quantified the contribution of the automatic embryo score and conventional morphological grade to implantation and live birth outcome with multivariate logistic regression analysis in different patient populations. A higher embryo score was associated with a better clinical outcome of IVF treatment. The mean of the automatic embryo score varied significantly (P < 0.001) among embryos with different morphological categories, between euploid and aneuploid embryos, between embryos resulting in positive versus negative pregnancy, between implanted and non-implanted embryos, and between embryos resulting in positive and negative live birth. Embryo score was related to the odds of implantation and live birth in the oocyte donation program (odds ratio (OR)=1.29; 95% CI [1.19-1.39]; P < 0.001 for implantation and OR = 1.26; 95% CI [1.16-1.36]; P < 0.001 for live birth) and in conventional treatments with autologous oocytes (OR = 1.38; 95% CI [1.24-1.54]; P < 0.001 for implantation and OR = 1.47; 95% CI [1.30-1.65]; P < 0.001 for live birth). There was no significant association of embryo score with implantation or live birth in treatments involving PGT-A. This study is limited by its retrospective nature. Further prospective randomized trials are required to confirm the clinical impact of these findings. The single-center design should be taken into account when considering the universal application of the model. Evidence of the clinical efficiency of automated embryo scoring for ranking embryos with different morphological grade and potential in order to achieve higher implantation and live birth rates may make it a decision support tool for embryologists when selecting blastocysts for embryo transfer. This research has been funded by a grant from the Ministry of Science, Innovation and Universities FIS (PI21/00283) awarded to M.M. There are no competing interests to declare. N/A.

A Method for Unsupervised Semi-Quantification of Inmunohistochemical Staining with Beta Divergences

In many research laboratories, it is essential to determine the relative expression levels of some proteins of interest in tissue samples. The semi-quantitative scoring of a set of images consists of establishing a scale of scores ranging from zero or one to a maximum number set by the researcher and assigning a score to each image that should represent some predefined characteristic of the IHC staining, such as its intensity. However, manual scoring depends on the judgment of an observer and therefore exposes the assessment to a certain level of bias. In this work, we present a fully automatic and unsupervised method for comparative biomarker quantification in histopathological brightfield images. The method relies on a color separation method that discriminates between two chromogens expressed as brown and blue colors robustly, independent of color variation or biomarker expression level. For this purpose, we have adopted a two-stage stain separation approach in the optical density space. First, a preliminary separation is performed using a deconvolution method in which the color vectors of the stains are determined after an eigendecomposition of the data. Then, we adjust the separation using the non-negative matrix factorization method with beta divergences, initializing the algorithm with the matrices resulting from the previous step. After that, a feature vector of each image based on the intensity of the two chromogens is determined. Finally, the images are annotated using a systematically initialized k-means clustering algorithm with beta divergences. The method clearly defines the initial boundaries of the categories, although some flexibility is added. Experiments for the semi-quantitative scoring of images in five categories have been carried out by comparing the results with the scores of four expert researchers yielding accuracies that range between and . These results show that the proposed automatic scoring system, which is definable and reproducible, produces consistent results.

Исследование применения графов знаний для генерации вопросов аудита информационной безопасности

The article presents approaches to research on the use of knowledge graphs to generate information security audit issues. The significance of the human factor in the audit of information security as an element of socially significant activity is shown. As a solution to the formation of an objective research base, a method of automatically generating questions based on knowledge graphs capable of identifying a set of cartoons that are subject-object relations is proposed. Existing models and a set of dynamically generated metrics were analyzed. In particular, the subjective method of the five-point Makoto Nagao scale and the machine method of the automatic evaluation system based on the N-gram method. Advantages and disadvantages of research methods are analyzed, task setting for further research is presented.

Automatized FACEmemory\xae scoring is related to Alzheimer\u2019s disease phenotype and biomarkers in early-onset mild cognitive impairment: the BIOFACE cohort

BackgroundFACEmemory® is the first computerized, self-administered verbal episodic memory test with voice recognition. It can be conducted under minimal supervision and contains an automatic scoring system to avoid administrator errors. Moreover, it is suitable for discriminating between cognitively healthy and amnestic mild cognitive impairment (MCI) individuals, and it is associated with Alzheimer’s disease (AD) cerebrospinal fluid (CSF) biomarkers. This study aimed to determine whether FACEmemory scoring is related to performance on classical memory tests and to AD biomarkers of brain magnetic resonance imaging (MRI) and CSF in patients with early-onset MCI (EOMCI).MethodsNinety-four patients with EOMCI from the BIOFACE study completed FACEmemory, classical memory tests (the Spanish version of the Word Free and Cued Selective Reminding Test -FCSRT-, the Word List from the Wechsler Memory Scale, third edition, and the Spanish version of the Rey–Osterrieth Complex Figure Test), and a brain MRI. Eighty-two individuals also underwent a lumbar puncture.ResultsFACEmemory scoring was moderately correlated with FCSRT scoring. With regard to neuroimaging MRI results, worse execution on FACEmemory was associated with lower cortical volume in the right prefrontal and inferior parietal areas, along with the left temporal and associative occipital areas. Moreover, the total FACEmemory score correlated with CSF AD biomarkers (Aβ1-42/Aβ1-40 ratio, p181-tau, and Aβ1-42/p181-tau ratio). When performance on FACEmemory was compared among the ATN classification groups, significant differences between the AD group and normal and SNAP groups were found.ConclusionsFACEmemory is a promising tool for detecting memory deficits sensitive to early-onset AD, but it also allows the detection of memory-impaired cases due to other etiologies. Our findings suggest that FACEmemory scoring can detect the AD endophenotype and that it is also associated with AD-related changes in MRI and CSF in patients with EOMCI. The computerized FACEmemory tool might be an opportunity to facilitate early detection of MCI in younger people than 65, who have a growing interest in new technologies.

Marking a collection of texts with the keywords – automation aspects

This article presents and discusses the results of automatic indexing of keywords in 27 functional collections of Russian texts in three functional styles: scholarly, journalistic and fiction. The approach to the processing of markup results is presented and the data on the consistency of experts are given. Depending on the nature of the project’s problems, the design tasks provide for an automated system of document and keyword identification. The aim of this study is to identify the problems of a modified automatic text scoring system (CAPT) with keywords and to analyse in detail the results of the scoring test in order to create the conditions for the next discourse. These functions constitute the content of one of the research stages aimed at creating an effective algorithm for CS extraction for Russian language.

Detection of pentylenetetrazol-induced seizure activity in the 19-21 Hz beta range using a magnetic coil induction method.

A force transducer or automatic scoring system is not sufficient to detect small or fine seizure activity. To improve previous assessments of epileptic behavior, a novel coil method was developed to detect an early behavioral marker for epileptic seizures. The present study used the γ-aminobutyric acid (GABA) receptor antagonist pentylenetetrazol (PTZ) to induce seizure activity and epileptic behavior in mice. A coil method was used to detect motor seizures consisting of small amplitude 19-21 Hz muscle contractions. Seizure activity in the 19-21 Hz range detected by the coil method was positively correlated with generalized clonic seizures with a kangaroo posture after PTZ administration. GABA receptor agonist valproic acid and ethosuximide decreased PTZ-induced 19-21 Hz seizure activity. The pattern of the amplitude ratio (%) of 19-21 Hz seizure activity after administration of the GABAA/C receptor antagonist picrotoxin was similar to the group that was treated with PTZ but different from the group that was treated with the nonselective muscarinic receptor agonist pilocarpine. The coil method detected 19-21 Hz seizure activity after PTZ administration. However, the force transducer method did not detect 19-21 Hz seizure activity. The coil method was more sensitive than the force transducer method for detecting epileptic behaviors. The findings may indicate a novel behavioral marker that can be detected by the coil method to reveal epileptic seizures, thus improving our understanding of the brain mechanisms of action and specific brain waves that are associated with PTZ-induced 19-21 Hz seizure activity.

Construction of artificial intelligence system of carpal bone age for Chinese children based on China‐05 standard

The purpose of this study is to construct an automatic carpal bone age evaluation system for Chinese children based on TW3-C Carpal method by deep learning and to evaluate the accuracies in test set and clinical test set. A total of 8184 radiographs of Chinese Han healthy children (2-14 years old) were collected from the 2005 China Skeletal Development Survey data and from the accumulated radiographs in bone age studies over the years. Three experienced radiologists and the China-05 standard maker jointly evaluated each bone development stage, and the consensual stage was decided as the reference standard. According to each epiphysis development stage, 10% of them were derived by stratified random sampling as test sets, and the remaining radiographs were used as training sets and validation sets. Furthermore, the overall performance of the model was estimated by comparing the mean difference, 95% limits of agreement, mean absolute difference (MAD) and root mean square (RMS) between the model predictions and the reference standard. The percentage agreement of ratings in each epiphysis in the test set ranged from 82.82% to 90.06%, with an average of 86.94%. Compared with the reference standard, the automated bone age system has a mean difference of 0.01 years old, ± 0.45 years old in 95% confidence interval by single reading, an 85.93% percentage agreement of ratings, a 90.5% bone age accuracy rate, 0.20 years old of MAD, and 0.32 years old of RMS in the clinical test set. The automatic bone age system for Chinese children has a comparable accuracy and stable determination compared with experienced radiologists.

Writing assistant scoring system for English second language learners based on machine learning

Abstract To reduce the workload of paper evaluation and improve the fairness and accuracy of the evaluation process, a writing assistant scoring system for English as a Foreign Language (EFL) learners is designed based on the principle of machine learning. According to the characteristics of the data processing process and the advantages and disadvantages of the Browser/Server (B/S) structure, the equipment structure design of the project online evaluation teaching auxiliary system is further optimized. The panda method is used to read the data, the clean method is used to realize the data preprocessing, the model test is carried out, the cross validation method is selected, the data is divided in advance, and the process of programming the problem scoring system is further optimized, the automatic scoring technology is constructed by English teaching recognition module, feature extraction module and scoring module, the table structure of programming problems is designed, the auxiliary evaluation program of English writing is designed, and the design of writing auxiliary scoring system is completed. The analysis of the experimental results shows that the accuracy of the system is close to 90%, and the total average difference is 0.56. The system can normally take out a variety of test papers. Considering the subjectivity of manual scoring and the impact of key code setting on scoring, the carefully set key code can effectively improve the scoring accuracy of the system. The scoring strategy of the automatic scoring system is effective and the scoring effect is good, and it can be used in practical application.

Automating Scoliosis Measurements in Radiographic Studies with Machine Learning: Comparing Artificial Intelligence and Clinical Reports.

Scoliosis is a condition of abnormal lateral spinal curvature affecting an estimated 2 to 3% of the US population, or seven million people. The Cobb angle is the standard measurement of spinal curvature in scoliosis but is known to have high interobserver and intraobserver variability. Thus, the objective of this study was to build and validate a system for automatic quantitative evaluation of the Cobb angle and to compare AI generated and human reports in the clinical setting. After IRB was obtained, we retrospectively collected 2150 frontal view scoliosis radiographs at a tertiary referral center (January 1, 2019, to January 1, 2021, ≥ 16years old, no hardware). The dataset was partitioned into 1505 train (70%), 215 validation (10%), and 430 test images (20%). All thoracic and lumbar vertebral bodies were segmented with bounding boxes, generating approximately 36,550 object annotations that were used to train a Faster R-CNN Resnet-101 object detection model. A controller algorithm was written to localize vertebral centroid coordinates and derive the Cobb properties (angle and endplate) of dominant and secondary curves. AI-derived Cobb angle measurements were compared to the clinical report measurements, and the Spearman rank-order demonstrated significant correlation (0.89, p < 0.001). Mean difference between AI and clinical report angle measurements was 7.34° (95% CI: 5.90-8.78°), which is similar to published literature (up to 10°). We demonstrate the feasibility of an AI system to automate measurement of level-by-level spinal angulation with performance comparable to radiologists.

Automatic Scoring System for Monitoring Foot Pad Dermatitis in Broilers

The assessment of foot pad dermatitis at slaughter is a suitable method to assess and monitor the welfare of broilers. The goals of this study were to define and validate a camera-based score that could identify macroscopic lesions of the foot pads, to identify errors, and to assess possible external factors that could influence the assessment. In the first phase 200 feet of broilers and in the second phase 500 feet were collected at slaughter, assessed visually, hung back into the evisceration line, and assessed by an automatic system. The camera score cut-off values were defined in the first (=calibration) phase. In the second (=validation) phase, the performance of diagnosis for these cut-off values was evaluated, and possible errors in the assessment of reference surface area and foot pad lesions were analyzed. The results showed that, in particular, Macro Scores 0, 2, and 3 could be identified with sufficiently high sensitivity. For Macro Score 1, the sensitivity of diagnosis was not sufficiently high in the two evaluated software versions. The current automatic assessment systems at slaughter could be adjusted to the cut-off values in order to classify foot pad dermatitis lesions. Furthermore, software updates can enhance the performance measures and lower the probability of errors.

An Analysis Model of Potential Topics in English Essays Based on Semantic Space

&lt;p&gt;With the reform of English examination in China in recent years, the automatic evaluation of English essays as subjective questions has always been the focus and difficulty of research. The existing automatic essay evaluation system (AEE) has obtained good feedback on the vocabulary, syntactic and other features of English essays, but there is still a problem of low accuracy in the analysis of potential topic in English essay. In order to solve this problem, this paper takes relational triples as the carrier to analyze the potential topics of English essays. By constructing the hierarchical topic trees hybrid semantic spaces to carry out topic clustering, distributed representation of topic relational triples and topic set extension in English essays. Then, based on the improved on-topic analysis algorithm in this paper, the paper analyzes the topic of English essay in multiple dimensions to obtain more abundant potential on-topic semantic information. The experiment results show that the proposed model can reduce the noise caused by non-topic words effectively, and improve the fine-grained topic semantic space in English essays, and the proposed model has better performance than the current methods of on-topic analysis in English esssays.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

Automatic Scoring of Spoken Language Based on Basic Deep Learning

The oral English test in domestic universities requires teachers to modify a large number of candidates’ oral recordings. This is the work of using time repeatedly. Using the CALL system to realize the automation of conversation recording can reduce the burden of teachers’ work. Therefore, it is of great practical significance to develop an automatic and accurate scoring system for oral English. With the development of artificial intelligence, deep learning technology has been gradually applied in various fields. Similarly, in the application of oral scoring, deep learning technology makes the implementation of such a system possible. Based on the deep learning technology, this paper proposes an automatic scoring algorithm for spoken language and implements a detailed design and evaluation system. The system consists of two modules. The pronunciation standard of spoken pronunciation and the content of spoken pronunciation are scored, and the sum of these two scores is the final score. Finally, this paper uses 650 oral English recordings from a college English test to train the artificial neural network. Experimental results show that if the training data set is small, the BP network model can obtain better comprehensive evaluation performance.