Automatic Quality Evaluation Research Articles

Research on the Construction of Teaching Quality Evaluation System

With the increasing application of information management methods to teaching management activities in Colleges and universities, it is imperative to use information system to realize the intelligent, automatic and accurate evaluation of teaching quality. This paper takes the evaluation system of teaching quality as the research object, focusing on the design and implementation of the evaluation system of teaching quality, so that we can evaluate the teaching performance of teachers fairly, reasonably and objectively from different perspectives such as students, leaders, supervision experts and colleagues, and fully mobilize the enthusiasm of teachers to complete the teaching work through the application of the system.

Automatic quality evaluation of parade by variance of postures of a platoon on single video camera logs

The highly synchronized parade can impress audiences how strong the troops seem to be, whereas it is difficult to train for the good parade because of its complex collective behavior. However, there is no scientific research about the important factor to train and produce a good parade is. One of the bottlenecks to the scientific approach is the difficulty of measurement of the quality of a group as same as other swarm researches. In this paper, we measured the posture data of members in the parade with OpenPose, which is a cutting-edge pose estimation technology of deep learning. By this measurement, we propose a numerical evaluation for the quality of the parade, and it is confirmed by the questionnaire. In conclusion, our evaluation method is applicable for the quantitative evaluation, and it was suggested that the variation level of the arm swing angles was related to the quality of the parade. This paper is based on the paper presented at the proceedings of the 3rd International Symposium on Swarm Behavior and Bio-Inspired Robotics [1].

Enhancement in Quality Estimation of Resistance Spot Welding Using Vision System and Fuzzy Support Vector Machine

The current nondestructive testing methods such as ultrasonic, magnetic, or eddy current signals, and even the existing image processing methods, present certain challenges and show a lack of flexibility in building an effective and real-time quality estimation system of the resistance spot welding (RSW). This paper provides a significant improvement in the theory and practices for designing a robotized inspection station for RSW at the car manufacturing plants using image processing and fuzzy support vector machine (FSVM). The weld nuggets’ positions on each of the used car underbody models are detected mathematically. Then, to collect perfect pictures of the weld nuggets on each of these models, the required end-effector path is planned in real-time by establishing the Denavit-Hartenberg (D-H) model and solving the forward and inverse kinematics models of the used six-degrees of freedom (6-DOF) robotic arm. After that, the most frequent resistance spot-welding failure modes are reviewed. Improved image processing methods are employed to extract new features from the elliptical-shaped weld nugget’s surface and obtain a three-dimensional (3D) reconstruction model of the weld’s surface. The extracted artificial data of thousands of samples of the weld nuggets are divided into three groups. Then, the FSVM learning algorithm is formed by applying the fuzzy membership functions to each group. The improved image processing with the proposed FSVM method shows good performance in classifying the failure modes and dealing with the image noise. The experimental results show that the improvement of comprehensive automatic real-time quality evaluation of RSW surfaces is meaningful: the quality estimation could be processed within 0.5 s in very high accuracy.

CycleGAN-based deep learning technique for artifact reduction in fundus photography.

A low quality of fundus photograph with artifacts may lead to false diagnosis. Recently, a cycle-consistent generative adversarial network (CycleGAN) has been introduced as a tool to generate images without matching paired images. Therefore, herein, we present a deep learning technique that removes the artifacts automatically in a fundus photograph using a CycleGAN model. This study included a total of 2206 anonymized retinal images including 1146 with artifacts and 1060 without artifacts. In this experiment, we applied the CycleGAN model to color fundus photographs with a pixel resolution of 256 × 256 × 3. To apply the CycleGAN to an independent dataset, we randomly divided the data into training (90%) and test (10%) datasets. Additionally, we adopted the automated quality evaluation (AQE) to assess the retinal image quality. From the results, we observed that the artifacts such as overall haze, edge haze, lashes, arcs, and uneven illumination were successfully reduced by the CycleGAN in the generated images, and the main information of the retina was essentially retained. Further, we observed that most of the generated images exhibited improved AQE grade values when compared with the original images with artifacts. Thus, we could conclude that the CycleGAN technique can effectively reduce the artifacts and improve the quality of fundus photographs, and it may be beneficial for clinicians in analyzing the low-quality fundus photographs. Future studies should improve the quality and resolution of the generated image to provide a more detailed fundus photography.

Automatic statistical evaluation of quality of unit selection speech synthesis with different prosody manipulations

Abstract Quality of speech synthesis is a crucial issue in comparison of various text-to-speech (TTS) systems. We proposed a system for automatic evaluation of speech quality by statistical analysis of temporal features (time duration, phrasing, and time structuring of an analysed sentence) together with standard spectral and prosodic features. This system was successfully tested on sentences produced by a unit selection speech synthesizer with a male as well as a female voice using two different approaches to prosody manipulation. Experiments have shown that for correct, sharp, and stable results all three types of speech features (spectral, prosodic, and temporal) are necessary. Furthermore, the number of used statistical parameters has a significant impact on the correctness and precision of the evaluated results. It was also demonstrated that the stability of the whole evaluation process is improved by enlarging the used speech material. Finally, the functionality of the proposed system was verified by comparison of the results with those of the standard listening test.

Usability evaluation of captions for people who are deaf or hard of hearing

Many Deaf and Hard-of-Hearing (DHH) individuals across the world benefit from various captioning services for accessing information existing in the form of speech. Today, the Automatic Speech Recognition (ASR) technology has the potential to replace the existing human-provided services for captioning due to their lowered cost of operation and ever-increasing accuracy. However, as with most automatic systems, ASR technology is still not fully perfect --- which leads to issues in terms of its trust and acceptance when focusing on building a human-free service of communication for these users. Thus, there is a need for evaluating the usability these systems with the users before deploying them into the real-world. Yet, most researchers lack access to sufficient DHH users for extrinsic, empirical studies of these automatic captioning systems. This articles presents our work on the development of automatic caption quality evaluation metric which we design and validate through studies and real-world observations with DHH users.

Dual Stage Image Analysis for a complex pattern classification task: Ham veining defect detection

Veins in pork thigh carcass are directly related to the quality of dry-cured ham, and consequently to its market value. Some veining defects over the surface of raw ham are easily detected by humans and precisely assessed by a specialist. However, the automatic evaluation of raw ham quality by image analysis poses some challenges to the traditional Computer Vision Systems (CVS), many of them grounded on the complex image pattern related to each defect level. To improve the CVS classification performance without overburdening feature extraction, as well as the common machine learning modelling, we propose Dual Stage Image Analysis (DSIA). DSIA is an additional step in a CVS, that was built in two stages based on the “divide and conquer” strategy. The first stage consists of splitting the region of interest into sub-regions to predict the presence of veining. In the second stage, the algorithm computes the number of veining sub-regions to assess the final defect level classification. A total of 194 raw ham samples were used to evaluate the DSIA performance in the experiments. Support Vector Machine and Random Forest algorithms were compared for inducing the classification model using 92 image features. Random Forest model was the best, capable of predicting defect level with 88.10% accuracy using DSIA. Without DSIA, the CVS with RF achieved an accuracy of 63.10%.

GF-3 Polarimetric Data Quality Assessment Based on Automatic Extraction of Distributed Targets

With the needs of continuous data quality assessment for massive Gaofen-3 (GF-3) polarimetric data, an automatic and efficient quality evaluation method is urgently needed. In this article, an automated polarimetric SAR data quality assessment method is conducted using a classic convolution neural network (VGG-16). The method is first pretrained, performance-tested, and robustness-tested on Radarsat-2 fully polarimetric data, then trained by selected SAR scenes of GF-3 for being applied on GF-3 data. The network is supposed to fulfill the work of automatically and accurately selecting those distributed targets satisfying quality evaluation under various scenes. A PolSAR data assessment method based on these distributed targets proposed by the authors in previous work is then applied to give the evaluation results. Experiments on GF-3 data and the comparison to prior works and corner reflectors on polarimetric distortion assessment results verify the effectiveness and advantages of the proposed method. The polarization data quality of GF-3 at different beams is also obtained. The technique and strategy in this article are practical and contributing to the long-term quality assessment of PolSAR data.

Developing a new mesh quality evaluation method based on convolutional neural network

One of the difficult requirements imposed on high-quality CFD mesh generation has been the ability to evaluate the mesh quality efficiently. Due to the lack of a general and effective evaluating criterion, the current mesh quality evaluation task mainly relies on various quality metrics for the shape of mesh elements, such as angle, radius, edge and contextual information collected by pre-processing software. However, this line of methods greatly increases the pre-processing cost and may not guarantee a precise quality result. In this paper, we provide a solution to solve the mentioned issues, resulting in a CNN model GridNet and the first mesh dataset NACA-Market. GridNet takes the mesh file as input and then automatically evaluates the mesh quality. Experiment results show that GridNet is capable of performing automatic mesh quality evaluation and outperforms the widely used classifiers. We hope that the proposed large benchmark collection and network could fill in the gaps in the fields of CNN-based mesh quality evaluation and provide potential future research directions in this field.

Paper] Automatic Quality Evaluation of Whole Slide Images for the Practical Use of Whole Slide Imaging Scanner

Paper] Automatic Quality Evaluation of Whole Slide Images for the Practical Use of Whole Slide Imaging Scanner

Automatic evaluation of geospatial data quality using web services

El sector geomático vive un escenario de sobrecarga de datos donde casi todos los días se generan nuevas bases de datos geoespaciales (BDG). Sin embargo, hay poca o ninguna información sobre la calidad de estas BDG. En este contexto proponemos una solución para la evaluación automática de la calidad de los datos geoespaciales utilizando servicios web. Este enfoque está compuesto por procedimientos de evaluación automática para el control de calidad de la consistencia topológica, comple- ción y exactitud posicional según se especifican en el estándar brasileño. Algunos procedimientos de control requieren datos externos para fines de comparación. Por ello, en este trabajo, proporcionamos un conjunto de datos sintéticos generados según un diseño de experimentos con el objetivo de seleccionar los métodos más adecuados para encontrar correspondencias entre las BDG. La solución desarrollada tiene un capa de interoperabilidad que vincula usuarios y procedimientos automáticos utilizando la interfaz del Web Processing Service (WPS). Nuestros resultados mostraron que el procedimiento automático funcionó muy similar al manual.

Evaluación automática de la calidad de datos geoespaciales mediante servicios web

The geomatics sector is going through a data overload scenario which new geospatial datasets are generated almost daily. However there are few or nothing infor- mation about the quality of these datasets, and they should be evaluated aiming to provide users some information about their quality. In this context we propose a solution for the automatic quality evaluation of geospatial datasets using the web services platform. This approach is compound by automatic evaluation procedures for quality control of topological consistency, completeness, and positional accuracy described in the Brazilian quality standard. Some procedures require an external dataset for comparison purposes. Hence we provide a set of synthetic datasets and apply over them an experimental design aiming to select suitable methods to find the correspondences between datasets. The solution has an interoperability tier that links users and automatic procedures using the standardized interface of Web Processing Service (WPS). Our results showed that the automatic procedure works very similar to the manual one.

Machine Learning for Automated Quality Evaluation in Pharmaceutical Manufacturing of Emulsions

In pharmaceutical industries, the quality assessment of emulsions is typically based on subjective examination of these samples under the microscope by trained analysts. The major drawbacks of such manual quality assessment include inter-observer variability, intra-observer variability, lack of speed and poor accuracy. In order to address these challenges, an automated approach, based on machine vision and machine learning, is investigated in this study. Micrographs, obtained during an emulsification process, are classified into four quality-based categories named TAMU (target, acceptable, marginal and unacceptable). A machine learning approach using principal component–based discriminant analysis is employed in this study for the classification. This approach is compared with manual classification results obtained for the same set of micrographs using attribute agreement analysis, which is a methodology of assessing the accuracy and precision of an evaluation system. The automated approach is demonstrated to be repeatable, 40% more accurate compared to the least performing analyst and 10% more accurate than the best performing analyst. The results show that the automated classification is superior to manual classification of micrographs with respect to speed (180 times faster), greater accuracy and repeatability. The automated approach, implemented as a soft sensor, integrated with real-time image acquisition can be applied for in situ process monitoring of emulsions. The real-time approach can be used to predict the instantaneous product quality as well as optimum process time required to achieve the desirable droplet characteristics, which will avoid over-processing and wastage of resources in pharmaceutical industries.

The First Step towards Automatic Quality Evaluation of Chinese Vowel Pronunciations for Foreign Learners for Self-training

The First Step towards Automatic Quality Evaluation of Chinese Vowel Pronunciations for Foreign Learners for Self-training

Оценка качества специализированной оториноларингологической помощи

The lecture is devoted to the content and assessment of the quality of medical care in general and specialized otorhinolaryngological care in particular. The lecture provides the basic concepts of medical care in general, its main types and forms of provision. The authors define the concept of specialized otorhinolaryngological care from the viewpoint of regulatory documents of the Ministry of Healthcare of Russia and current regulatory documents, its role in the system of medical care of our country, providing the routing of the patients at all its stages. The lecture presents the approaches to assessing the quality of specialized otorhinolaryngological care. The authors determine the concept of mistake and defect, their main types and causes, their effect on the safety and efficacy of specialized otorhinolaryngological care. The work covers the legal aspects of assessment of provided otorhinolaryngological care quality, pointing out the important problem of the absence of a single approach to such assessment in otorhinolaryngology. The authors substantiate the necessity of original and multipurpose system of expert assessment of quality of provided otorhinolaryngological care based on automatic quality evaluation technology as a prototype. The next work will be devoted to its peculiarities as well as to the problems existing in clinical otorhinolaryngology.

242. A new approach for MRI 3D image quality control: Protocol set up for multicoil arrays

Purpose The use of RF multicoil arrays for MR imaging has demonstrated several advantages, however when the number of coil elements increases, both signal and noise can become very inhomogeneous over the scanned volume, so standard 2D evaluations, as proposed by many protocols, are no longer representative of the whole volume and a new approach has to be considered. In this work we set up an automatic 3D quality evaluation particularly suited for head coils. Methods Measurements were performed on a Siemens Skyra 3T system with a 20 channel TX head coil, scanning an homogeneous spherical phantom with a standard clinical 3D protocol. Reconstruction was performed both for single sub-coil (uncombined images) and for the final image volume, that is a combination of all the sub-coils signals. Processing was performed offline with a bespoke imagej plugin. Main processing steps for combined volume are the automatic centring of a spherical region of interest (ROI) covering 80% of the phantom image volume, and the calculation of Signal to Noise Ratio (SNR), Normalized Absolute Average Difference (NAAD) and 5-color level reconstructed image with the NEMA algorithms for 2D images that were extended to 3D [1] . For uncombined sub-coil images, SNR is calculated in a 2 cm diameter sphere ROI automatically centered on the maximum average signal region, as showed in Fig. 1, and uniformity is evaluated with a 11-color levels volume reconstruction. Results Results for volumetric analysis are compared to the evaluation performed on 2D images from standard planes centered at the phantom isocenter. Results are very different particularly for uncombined analysis where the single coil is far from the evaluation slice, as reported for example in Table 1 for the SNR of coil n.2. In this case parameter variations due to malfunctioning are less sensitive with a 2D analysis. Conclusions We set up an automatic protocol for 3D quality evaluation which is more representative of the whole coil specification volume and more sensitive to failures when a single coil analysis is performed.

Human versus automatic quality evaluation of NMT and PBSMT

Neural machine translation (NMT) has recently gained substantial popularity not only in academia, but also in industry. For its acceptance in industry it is important to investigate how NMT performs in comparison to the phrase-based statistical MT (PBSMT) model, that until recently was the dominant MT paradigm. In the present work, we compare the quality of the PBSMT and NMT solutions of KantanMT—a commercial platform for custom MT—that are tailored to accommodate large-scale translation production, where there is a limited amount of time to train an end-to-end system (NMT or PBSMT). In order to satisfy the time requirements of our production line, we restrict the NMT training time to 4 days; to train a PBSMT system typically requires no longer than one day with the current training pipeline of KantanMT. To train both NMT and PBSMT engines for each language pair, we strictly use the same parallel corpora and the same pre- and post-processing steps (when applicable). Our results show that, even with time-restricted training of 4 days, NMT quality substantially surpasses that of PBSMT. Furthermore, we challenge the reliability of automatic quality evaluation metrics based on n-gram comparison (in particular F-measure, BLEU and TER) for NMT quality evaluation. We support our hypothesis with both analytical and empirical evidence. We investigate how suitable these metrics are when comparing the two different paradigms.

QUALITY EVALUATION OF LAND-COVER CLASSIFICATION USING CONVOLUTIONAL NEURAL NETWORK

Abstract. Land-cover classification is one of the most important products of earth observation, which focuses mainly on profiling the physical characters of the land surface with temporal and distribution attributes and contains the information of both natural and man-made coverage elements, such as vegetation, soil, glaciers, rivers, lakes, marsh wetlands and various man-made structures. In recent years, the amount of high-resolution remote sensing data has increased sharply. Accordingly, the volume of land-cover classification products increases, as well as the need to evaluate such frequently updated products that is a big challenge. Conventionally, the automatic quality evaluation of land-cover classification is made through pixel-based classifying algorithms, which lead to a much trickier task and consequently hard to keep peace with the required updating frequency. In this paper, we propose a novel quality evaluation approach for evaluating the land-cover classification by a scene classification method Convolutional Neural Network (CNN) model. By learning from remote sensing data, those randomly generated kernels that serve as filter matrixes evolved to some operators that has similar functions to man-crafted operators, like Sobel operator or Canny operator, and there are other kernels learned by the CNN model that are much more complex and can’t be understood as existing filters. The method using CNN approach as the core algorithm serves quality-evaluation tasks well since it calculates a bunch of outputs which directly represent the image’s membership grade to certain classes. An automatic quality evaluation approach for the land-cover DLG-DOM coupling data (DLG for Digital Line Graphic, DOM for Digital Orthophoto Map) will be introduced in this paper. The CNN model as an robustness method for image evaluation, then brought out the idea of an automatic quality evaluation approach for land-cover classification. Based on this experiment, new ideas of quality evaluation of DLG-DOM coupling land-cover classification or other kinds of labelled remote sensing data can be further studied.

Non-destructive automatic quality evaluation of fresh-cut iceberg lettuce through packaging material

Non-destructive evaluation of vegetables by Computer Vision Systems (CVSs) makes possible to check their quality level in an objective and consistent way along the whole supply chain up to the final users. CVSs have been proven to be successful when applied to unpackaged products.The proposed approach aimed to enable this analysis on packaged fresh-cut lettuce with minimum constraints on the acquisition phase and without any care to flatten the surface of the bag facing the camera. A deep-learning architecture, based on Convolutional Neural Networks (CNNs), was used to identify regions of the image where the vegetable was visible with minimum colour distortions due to packaging. To meaningfully assess the performance of the system, each lettuce's sample was acquired both through packaging material and without packaging material. The image analysis was applied to both the resulting images to automatically grade their quality level. The results showed that the performance loss due to the presence of packaging is negligible (83% instead of 86%) and that the proposed system can be used to monitor the quality level of fresh-cut lettuce regardless of packaging at all the critical check points along the supply chain.

A retinal image sharpness metric based on histogram of edge width

Retinal image sharpness assessment is one of the critical requirements of automatic quality evaluation in telemedicine screening for diabetic retinopathy. In this paper, a new sharpness metric measuring the spread of edges is presented to quantify fundus image clarity. After edge detection on the region of interest of retinal image, the width of each edge is calculated and the histogram of region of interest generated. Based on the histogram, a distance-based factor is introduced to gain the weighted edge width, which is defined as the sharpness metric for the fundus image. The method was tested on Messidor dataset and a proprietary dataset. The results show that the proposed metric performs well over different image distortion levels and resolutions and is of low computational complexity. The weighted edge width value of gradable retinal image, which is irrelevant to resolution, is always within the range of 3–7 pixels.