Automated Scoring System Research Articles

Quantitative lung ultrasonography: a putative new algorithm for automatic detection and quantification of B-lines

BackgroundThis pilot study was designed to develop a fully automatic and quantitative scoring system of B-lines (QLUSS: quantitative lung ultrasound score) involving the pleural line and to compare it with previously described semi-quantitative scores in the measurement of extravascular lung water as determined by standard thermo-dilution.MethodsThis was a prospective observational study of 12 patients admitted in the intensive care unit with acute respiratory distress and each provided with 12 lung ultrasound (LUS) frames. Data collected from each patient consisted in five different scores, four semi-quantitative (nLUSS, cLUSS, qLUSS, %LUSS) and quantitative scores (QLUSS). The association between LUS scores and extravascular lung water (EVLW) was determined by simple linear regression (SLR) and robust linear regression (RLR) methods. A correlation analysis between the LUS scores was performed by using the Spearman rank test. Inter-observer variability was tested by computing intraclass correlation coefficient (ICC) in two-way models for agreement, basing on scores obtained by different raters blinded to patients’ conditions and clinical history.ResultsIn the SLR, QLUSS showed a stronger association with EVLW (R2 = 0.57) than cLUSS (R2 = 0.45) and nLUSS (R2 = 0.000), while a lower association than qLUSS (R2 = 0.85) and %LUSS (R2 = 0.72) occurred. By applying RLR, QLUSS showed an association for EVLW (R2 = 0.86) comparable to qLUSS (R2 = 0.85) and stronger than %LUSS (R2 = 0.72). QLUSS was significantly correlated with qLUSS (r = 0.772; p = 0.003) and %LUSS (r = 0.757; p = 0.005), but not with cLUSS (r = 0.561; p = 0.058) and nLUSS (r = 0.105; p = 0.744). Moreover, QLUSS showed the highest ICC (0.998; 95%CI from 0.996 to 0.999) among the LUS scores.ConclusionsThis study demonstrates that computer-aided scoring of the pleural line percentage affected by B-lines has the potential to assess EVLW. QLUSS may have a significant impact, once validated with a larger dataset composed by multiple real-time frames. This approach has the potentials to be advantageous in terms of faster data analysis and applicability to large sets of data without increased costs. On the contrary, it is not useful in pleural effusion or consolidations.

32. IMPROVING EFFICIENCY OF AUTISM SCREENING

Background Autism is a neurodevelopmental disability appearing before age 3. Children with autism have difficulties in verbal and non-verbal communication, social interaction and play activities. M-CHAT is a 20-question parental survey about a child's social, verbal and emotional development. It is easy to use and validated in primary care settings with 85% sensitivity and 93% specificity. The AAP recommends that all children be screened for autism at ages 18 and 24 months, along with regular developmental surveillance. In Children's Health Center of The Brooklyn Hospital Center, we use Office Practicum with an automated scoring system of M-CHAT. Residents ask 20 questions during the encounter and document answers in the EMR as routine practice at 18 and 24 months visits. We propose a different approach using an M-CHAT paper form to be filled out by the parent before the encounter to improve efficiency of autism screening in a busy primary care clinic. Aim To decrease M-CHAT screening time by 50% and achieve a 100% completion rate at 18 and 24 months visits by May 2018 Interventions PDSA cycle I: Residents ask parent to fill out M-CHAT before the encounter of 18 and 24 months visits. Residents review the answers and input them into the EMR with automated scoring system. Residents address any concerns parent might have. PDSA cycle II: M-CHAT handout is available in front desk with multiple language versions. Educate the front desk staff to distribute M-CHAT to parent at 18 or 24 months well child visit after registration. Measures Autism screen time using M-CHAT-R; Completion rate of documented M-CHAT in EMR at 18 and 24 months visit Results During PDSA cycle I, a significant reduction of autism screen time was noted, with the median time being 1 minute compared to 4.5 minutes without intervention. There were 2 patients with a positive screen result and the screen times were 5 and 10 minutes respectively, which we excluded from data. The average time spent in M-CHAT assessment with negative screen result continued to be 1 minute in PDSA cycle II. In the pre-intervention study from October to December 2017, the documented M-CHAT completion rate was 80% at 18 months visits and 78% at 24 months visits. In January 2018 (PDSA cycle I), the M-CHAT completion rate increased to 100% at 18 months visits and 94.7% at 24 months visits. In PDSA cycle II, the M-CHAT completion rate was 94.6% at 18 months visits and 90% at 24 months visits. Yet, the rate of M-CHAT distribution by front desk staff was 64.5%. Conclusions and Next Steps This quality improvement study demonstrated significant reduction of autism screening time and improvement of workflow, and positive changes in completion of the M-CHAT documented on 18 and 24 months well child visits. Our intervention process excluded child with positive screens due to additional discussion regarding proper follow-up and referral. Use of the M-CHAT handout before the encounter markedly increased efficiency of autism screening, yet adaption to this practice among clinic staff likely contributed to the incompletion rate in quality metrics. Overall, we found the use of M-CHAT paper forms before the encounter helped residents incorporate information faster and assess the risk for autism more efficiently. As a result of this study, the practice leadership decided to continue using the processes introduced in this intervention.

Apple growth evaluated automatically with high-definition field monitoring images

During cultivation of agricultural crops, it is important to accurately evaluate crop growth and perform the appropriate work at the right time. However, because a system for inexpensively and accurately determining crop growth automatically over time has not been developed, it is generally difficult for farmers to decide when to perform agricultural work. Therefore, we developed a system that can inexpensively and accurately evaluate the growth of crops over time. First, we developed a high-definition image monitoring device and periodically took images of an apple tree from a fixed point for 2 years from 2016 to 2017. Then, to obtain apple growth information, the radii of the apples in the images were measured manually, and the average radius was standardized from data for both years. We assumed that metrics for evaluating apple growth could be calculated from high-definition field monitoring images of the apple tree. Then, we introduced the green–blue vegetation index (GBVI), which can be easily calculated from images taken with a visible light camera installed near an apple tree, and calculated the GBVI leaf area. Then, we analyzed the metrics that can evaluate apple growth. Specifically, logistic curve approximations were performed for both years with a standardized average apple radius as the dependent variable, and the effective cumulative temperature, cumulative sunshine duration, cumulative precipitation, cumulative maximum GBVI leaf area, cumulative average GBVI leaf area, and elapsed time were taken as independent variables. The obtained logistic curves were defined as growth curves. The results showed that when the cumulative maximum GBVI leaf area was used as an independent variable, the growth curves were almost the same in both years. Moreover, as a result of estimating the standardized average apple radius using each growth curve, the estimation error for the cumulative maximum GBVI leaf area used as an independent variable was the smallest among all the independent variables. These results suggested that the cumulative maximum GBVI leaf area calculated from images was the most useful metric for accurately evaluating apple growth. In addition, it was suggested that the harvestable apple radius could be predicted using the growth curve with the cumulative maximum GBVI leaf area as the independent variable. By incorporating the methods for calculating these metrics into the high-definition image monitoring device, we can develop an automatic apple growth evaluation system based on high-definition field monitoring images.

Multi-attribute decision-making method for prioritizing maritime traffic safety influencing factors of autonomous ships’ maneuvering decisions using grey and fuzzy theories

Ship maneuvering decisions are influenced by several factors, and it is essential to prioritize the main influencing factors for efficient selection of the corresponding maneuvering decisions. Meanwhile, the autonomous ship maneuvering decision-making influencing factors constitute a typical grey system, which is suitable for research by grey relational analysis. Furthermore, in the fuzzy approach, linguistic assessment of factors is evaluated to obtain priorities numbers. Therefore, this study mainly focuses on the concept of human-like maneuvering for autonomous ships. Based on experimental data of experienced seafarers and using a simulation platform under the scenario of the Shanghai Waigaoqiao wharf, an inference model utilizing grey and fuzzy theories is proposed. The proposed model combined with expert linguistic terms in order to select the ship maneuvering decision-making main influencing factors from multi-source influencing factors (in overall and separated categories of natural environment, ship motion, force parameters, draft, and position), and to study the decision-making prioritization for maritime traffic safety for specific ship maneuvering scenarios. This method can prioritize the main factors which affect maneuvering decisions as well as guide an autonomous ship-assisted or automatic maneuvering evaluation system for the research of human-like maneuvering behavior. This study provides a new perspective on the identification of main ship maneuvering decision-making influencing factors in theory and in practice. It can be utilized for better decision-making concerning maritime traffic safety of autonomous ship maneuvering, which in turn makes shipping safer and promote the application and spreading of autonomous ships.

Construction of Automatic Scoring System to Support Objective Evaluation of Clinical Skills in Medical Education.

In this study, we focused on the automatic scoring of medical clinical abilities. The objective clinical ability tests that all undergraduate students take before starting clinical practice were considered. As these tests evaluate practical skills, there is a problem that the learning method is poor compared to the examination of other lectures. Therefore, in this study, we recorded the voice of a student examining a simulated patient using a microphone. We constructed a system comprising a speech recognition module and a scoring system that performed automatic scoring by checking against a prepared example answer. This system was evaluated by medical doctors.

Task-Based Automatic Evaluation of People with Intellectual Disabilities Performed on a Robotic Table Soccer.

This paper is concerned with the automatic evaluation of selected tasks performed by people with intellectual Disabilities. According to the International Classification of Functioning, Disability and Health (ICF) system, subjects must be divided into two groups: group with no difficulty (N) and group with difficulty (D) being this classification based on performances obtained in a conventional table (CT) soccer. Three tasks, with different levels of difficulty, were proposed for performance evaluation. Experimental results were obtained on the basis of the task execution in both a CT and a robotics table (RT) soccer. All participants were able to perform tasks with the joystick on the RT soccer and the automatic evaluation system identified differences in reaction times with and without red color flag in the participants, on RT soccer. One of the tasks was completely performed by all the participants by using the RT soccer.

A Study of Applying Automated Assessment in Teaching College English Writing Based on Juku Correction Network

With the continuous advancement of modern network technology, the drawbacks of the tradition-al English writing course teaching mode have become increasingly prominent, and the automated scoring system has gradually been used in the writing course. This paper proposes a college English writing teaching model based on Juku Correction Network, and conducts empirical re-search on the use of Juku Correction Network in college English writing teaching. The research results show that the teaching mode based on Juku Correction Network can effectively improve the overall level of students' English writing, and stimulate students' English writing motivation.

Improving human scoring of prosody using parametric speech synthesis

This paper proposes a method that utilizes parametric speech synthesis to improve human scoring of non-native speaker utterances. Instead of assessing each prosodic feature by directly listening to the utterance itself, in order to focus only on the target prosodic feature, the unassessed features are substituted with those of the native speakers. We used parametric speech synthesis to generate the features for substitution. In this study, HMM-based speech synthesis from an average model of native speakers was utilized. The experimental result shows that the proposed method can improve scoring reliability, which is confirmed by an increase in the inter-rater correlation. We also build an automatic pronunciation evaluation system trained from non-native speech databases with scores given by either the conventional and proposed methods, and compare the performance of the systems. The result shows that the predicted pronunciation scores matched the human-rated scores; the human-machine correlation produced a score of 0.87, while the conventional scoring method produced a score of 0.74.

LSTM と Attention を用いた自動採点及び採点支援の実用化に向けて

LSTM と Attention を用いた自動採点及び採点支援の実用化に向けて

Language History Questionnaire (LHQ3): An enhanced tool for assessing multilingual experience

AbstractThe language history questionnaire (LHQ) is an important tool for assessing the linguistic background and language proficiency of multilinguals or second language learners. Previously we developed a generic LHQ based on the most commonly asked questions in published studies (Li, Sepanski & Zhao, 2006) and provided a web-based interface (LHQ 2.0) that has flexibility in functionality, accuracy in data recording, and privacy for users and data (Li, Zhang, Tsai & Puls, 2014). LHQ3 (version 3) introduces new functions, developed in response to many comments/requests from users. One important improvement is the addition of an automatic scoring system, in that the new interface automatically calculates aggregated scores for language proficiency, language dominance, and language immersion levels. Finally, LHQ3 allows researchers to assign different weights to the modules when calculating the aggregated scores, addressing the issue of different focuses that different researchers put on multilingual speakers’ language usage and background.

Mega Hpv laboratories for cervical cancer control: Challenges and recommendations from a case study of Turkey

Cervical cancer is the fourth most common cancer among women in the world. It is estimated that one woman dies every 2 min from cervical cancer. Nearly all cervical cancers are preventable by early detection and treatment through screening or HPV vaccination. In 2018, World Health Organization (WHO) made a global call for action toward the elimination of cervical cancer. Cervical cancer screening involves a complex organized program, which begins with a call/recall system based on personal invitation of eligible women, followed by participation in screening, and leading to diagnosis, treatment, and management as appropriate. An effective cervical screening program with high coverage is dependent on each country's infrastructure and human resource capacity. Efforts to develop an effective program is particularly challenging in low and middle income countries (LMIC) where resources are limited. For an effective strategy, Turkey redesigned the country's cervical screening program. The local call/recall system and centralized monitoring system of individual women were re-vamped with an automated evaluation system. The revised screening program includes the use of primary HPV testing with a well-defined protocol outlining the algorithms of management (i.e., screening intervals and referral), a single nationwide centralized diagnostic laboratory, and a sustainable agreement with the HPV diagnostics industry. This system allows for traceable, real-time monitoring of screening visits and specimens. Turkey reports on the first four years of this re-vamped organized program and shares lessons learnt from the implementation of this new program.

Implementation and Evaluation of Algorithms with ALGator

In this paper we present an automatic algorithm evaluation system called ALGator, which was developed to facilitate the algorithm design and evaluation process. The system enables unbiased tests of the correctness of the algorithm's results on given test cases and comparisons of the quality of implemented algorithms for solving various kinds of problems (e.g. sorting data, matrix multiplication, traveler salesman problem, shortest path problem, and the like). Within the ALGator one can define a problem by specifying the problem descriptors, test sets with corresponding test cases, input parameters and output indicators, algorithm specifications and criteria for measuring the quality of algorithms. When a user of the system submits an algorithm for solving a given problem, ALGator automatically executes this algorithm on predefined tests, measures the quality indicators and prepares the results to be compared with the results of other algorithms in the system. The ALGator is meant to be used by algorithm developers to perform independent quality tests for their solutions.

Converting the DriveSafe subtest of DriveSafe DriveAware for touchscreen administration.

DriveSafe measures awareness of the driving environment. It is one subtest of DriveSafe DriveAware, a cognitive fitness-to-drive screening instrument. We converted DriveSafe to a touchscreen format for ease of administration; this necessitated the development of an automatic data collection and scoring system to reflect the decision that would otherwise have been made by an expert rater. We applied a structured process to determine what constituted 'correct' scores. We then examined the resulting scoring parameters to determine if these discriminated at-risk drivers from a comparison sample. Thirty at-risk older drivers and 30 younger drivers took touchscreen DriveSafe. Following presentation of images containing between one and four objects/hazards for 4 seconds, participants indicated their recall of object/hazard characteristics (type, location and direction of movement) by touching the screen. We analysed responses via descriptive statistics to compare spread, accuracy and consistency; and via a Fisher's exact test to determine whether the set scoring parameters could discriminate between at-risk and comparison drivers. Fisher's exact test results indicated 24 of 28 location zones and 18 of 28 direction ranges discriminated at-risk drivers from the comparison group (P < 0.05). Frequency of missed or incorrectly identified hazards was much higher for the at-risk group for all variables. At-risk drivers missed or misidentified significantly (P < 0.00) more object types (34%), directions (47%), and locations (36%) than the comparison group (≤4% for each variable). At-risk drivers entered 31 additional responses for objects/hazards not displayed; the comparison group entered no additional responses. The automatic variable data collection and scoring system reflected decisions that would have been made by an expert rater. This systematic process provided automated scoring decisions that enabled us to discriminate at-risk drivers from a comparison group. Psychometric evaluation of data gathered with touchscreen DriveSafe is required prior to use in clinical practice.

Meta-analysis of the Diagnostic Test Accuracy of Pediatric Inpatient Fall Risk Assessment Scales.

PurposeThis study was conducted to obtain data for the development of an effective fall risk assessment tool for pediatric inpatients through a systematic review and meta-analysis of the diagnostic test accuracy of existing scales.MethodsA literature search using Medline, Science Direct, CINAHL, EMBASE, and the Cochrane Library was performed between March 1 and 31, 2018. Of 890 identified papers, 10 were selected for review. Nine were used in the meta-analysis. Stata version 14.0 was used to create forest plots of sensitivity and specificity. A summary receiver operating characteristic curve was used to compare all diagnostic test accuracies.ResultsFour studies used the Humpty Dumpty Falls Scale. The most common items included the patient's diagnoses, use of sedative medications, and mobility. The pooled sensitivity and specificity of the nine studies were .79 and .36, respectively.ConclusionConsidering the low specificity of the pediatric fall risk assessment scales currently available, there is a need to subdivide scoring categories and to minimize items that are evaluated using nurses' subjective judgment alone. Fall risk assessment scales should be incorporated into the electronic medical record system and an automated scoring system should be developed.

Automated Management System for Accreditation of Clinical Engineering Department in Hospitals

The primary objective of clinical engineering department (CED) in hospitals is to provide a broad-based engineering program that addresses all aspects of healthcare technology. However, accreditation of this department is rarely considered in the literature. In this study, we attempted to identify systematic differences in quality management between the Joint Commission International (JCI)–accredited and non–JCI-accredited CED by designing a specific automated evaluation system depending on the technology of cloud computing. Understanding the similarities, dissimilarities, and factors that sustain accreditation within hospitals is the main aim of this work. Another goal is to investigate, describe, and prioritize the measurements and requirements demanded to achieve a minimum accepted level that qualifies nonaccredited ones to be JCI accredited. A questionnaire that targeted the clinical engineering department was developed. It comprises CED roles and the corresponding JCI standards to differentiate accredited from nonaccredited CEDs. A score index has been developed using answering rates of questionnaire to assess CED roles and overall performance of the department. Based on the score index, the CED is classified into 4 classes. System verification was carried out using a data set of 9 Egyptian hospitals. The results revealed the minimum benchmark level of acceptance for JCI accreditation. The system proves its consistency and correctly separated JCI-accredited from nonaccredited CEDs.

Difference Based Metrics for Deep Reinforcement Learning Algorithms

Although state-of-the-art deep reinforcement learning often achieves superhuman levels on some tasks, the authors still struggle to analyze, compare or report the obtained results due to the unstable nature of the algorithms and the diversity of metrics used in the literature. Furthermore, these metrics fail to show some characteristics of the learning process, leading to misinterpretations by the analyst. The objective of this paper is to propose, implement and analyze a difference based evaluation metric that highlight different aspects of the learning process, allowing for more detailed results and analysis that can be used by automated software or analysts, experienced or not. A possible applicability for the proposed metric is to create automated evaluation systems, that can detect anomalies during the training process automatically.

Evaluation of hemiplegia caused by stroke by using joint detection of depth sensors—case of SIAS

AbstractStroke Impairment Analysis Set (SIAS) is a set of evaluation methods of the hemiplegia body caused by stroke. The score of the test is made by visual examiners. However, the decisions are often different between them. Thus, the purpose of this study is to make an automatic evaluation system for the SIAS by using depth sensors. Kinect and Leap Motion are used to evaluate the proximal and distal upper extremity tests and the proximal lower extremity test in SIAS. The evaluation of these tests is based on the time difference of left and right cases, and the smoothness. If both of left and right motions are roughly the same and the motion is smooth enough, the score is 5, otherwise the score is decreased to between 4 and 0. The perfect match ratio between the system output and therapist judged is 65‐75% and it can be higher than this if similar judgments are partially added to the score. In experimental results, this system has been shown to yield reasonable scores for the knee‐mouth test, the finger test and the hip flexion test in motor function of SIAS.

IMPLEMENTATION OF GENERALIZED VECTOR SPACE MODEL METHOD AT AUTOMATIC ASSESSMENT OF ONLINE ESSAY EXAM

The aim of the study was to design and implement automatic testing of online essay examinations using the Generalized Vector Space Model (GVSM) method. This data is obtained through (1) Literature Study (2) Observation (3) Documentation. The results of this study indicate that the automatic scoring system with the GVSM weighting method and the cosine similarity similarity calculation method have the accuracy of the assessment with an average of 66%.

356. ZeroDose: An automatic tool for exposure indexes retrieval from images stored in PACS system

Purpose The evaluation of radiation exposure of patients undergoing diagnostic imaging is more than ever an incoming issue. As direct consequence of the forthcoming transposition of the EU59/2013 our division is exploring the use of an automatic dose evaluation system. Methods The system can be summarized in three phases: retrieve DICOM images, extrapolate the exposure indexes and analyze them. The retrieval of DICOM images from IMPAX, (AGFA, Healthcare) PACS system, was possible thanks to the freely available Conquest-DICOM-Software. In order to extract CT exposure indexes and acquisition parameters from DICOMtags we used ZeroBrane-Studio, an Integrated-Development-Environment, to write a LUA script that automatically reads and stores tag values. The script could be launched in batch mode and run over a predefined list of exams. The output is a csv file that contains all the extracted data and that could be analyzed with statistical tool R software. The system was tested on a set of patient (object of a separate abstract) for whom all exposure indexes and acquisition parameters were manually extracted from DICOMtags. Results All the CT, performed on quite modern machines, provided the correct information. The system had problem of DICOM compatibility with an old CT that was dismissed in 2015, the CTDIvol were not available at all on DICOM images. The processing time for a day of CT examination at S. Croce and Carle (85 exams) was about 10 min. We are adapting the script to PET modality. Conclusions This preliminary experience demonstrated that the tool could be safely used to extract information from DICOM images and direct them to text file that could be used for analysis and for reporting to a RIS system. We choose Conquest as DICOM server since our division has over 15-years’ experience on its use for the transfer and analysis of images.

Towards Evaluating Narrative Quality In Student Writing

This work lays the foundation for automated assessments of narrative quality in student writing. We first manually score essays for narrative-relevant traits and sub-traits, and measure inter-annotator agreement. We then explore linguistic features that are indicative of good narrative writing and use them to build an automated scoring system. Experiments show that our features are more effective in scoring specific aspects of narrative quality than a state-of-the-art feature set.