Automatic Evaluation Method Research Articles

An automatic evaluation method for retinal image registration based on similar vessel structure matching.

Registration of retinal images is significant for clinical diagnosis. Numerous methods have been proposed to evaluate registration performance. The available evaluation methods can work well in normal image pairs, but fair evaluation cannot be obtained for image pairs with anatomical changes. We propose an automatic method to quantitatively assess the registration of retinal images based on the extraction of similar vessel structures and modified Hausdorff distance. Firstly, vessel detection and skeletonization are performed to detect the vascular centerline. Secondly, the vessel segments having similar structures in the image pair are selected for assessment of registration. The bifurcation and terminal points are determined from the vascular centerline. Then, the Hungarian matching algorithm with a pruning process is employed to match the bifurcation and terminal points to detect similar vessel segments. Finally, a modified Hausdorff distance is employed to evaluate the performance of registration. Our experimental results show that the Pearson product-moment correlation coefficient can reach 0.76 and 0.63 in test set of normal image pairs and image pairs with anomalies respectively, which outperforms other methods. An accurate evaluation can not only compare the performance of different registration methods but also can facilitate the clinical diagnosis by screening out the inaccurate registration. Graphical abstract .

Adaptation of Full-Reference Image Quality Assessment Methods for Automatic Visual Evaluation of the Surface Quality of 3D Prints

Automatic visual quality assessment of the 3D printed surfaces is currently one of the most demanding challenges in additive manufacturing. Regardless of the applications of the computer vision for the 3D printing process monitoring purposes, a reliable surface quality evaluation during manufacturing may introduce brand new possibilities. The detection of some distortions and their automatic evaluation can be helpful when deciding to stop the process to save time, energy, and filament. In some cases, some further corrections can also be made for relatively small distortions. Since many general-purpose image quality assessment methods have been proposed in recent years, their applications for the quality evaluation in the additive manufacturing are investigated. As most of the metrics are full-reference and require the availability of the original perfect quality image, their direct application is not possible. Therefore, their adaptation is described in the paper together with experimental verification of classification results obtained using various metrics.

Impact of aging on semiquantitative uptake parameters in normal rated clinical baseline [123I]Ioflupane single photon emission computed tomography/computed tomography.

Studies investigating the age-related impact on dopamine transporter binding have previously omitted the use of attenuation correction by computed tomography (CT). We aimed to explore the impact of age and gender on dopamine transporter binding on [I]Ioflupane single photon emission CT (SPECT) imaging with simultaneously acquired CT. Three hundred forty-two patients with clinically uncertain parkinsonian syndrome underwent [I]-Ioflupane SPECT/CT with CT-based attenuation correction. Two nuclear medicine physicians independently performed a visual evaluation of all scans and only visibly normal scans were included for further analysis. Moreover, the results of a fully automatic semiquantitative evaluation method were recorded. Thereafter, the obtained [I]Ioflupane binding ratio and the hemispheric asymmetry index were correlated with age and sex. Patient age range was 41-80 years with a balanced distribution over decades. Of 342 patients, 133 (38.9%, 66 females, median age, 64 years) were considered visually normal by both observers on the SPECT/CT images. A significant inverse correlation between age and [I]Ioflupane binding ratios in the striata (R = -0.38; P < 0.001), putamina (R = -0.39; P < 0.001) and caudate nuclei (R = -0.3; P < 0.001) was demonstrated. Linear regression of all included subjects demonstrated an average decrease of 0.19 per decade in the striatal binding ratio (6.6%). No significant sex differences were found in striatal binding ratios (P = 0.86). Moreover, no significant correlation was observed between age and striatal asymmetry index (r = 0.12; P = 0.16). In the present largest single-center analysis investigating [I]Ioflupane SPECT/CT in patients with clinical uncertain parkinsonian syndrome, a dopamine transporter loss of 6.6% per decade in visually normal scans was recorded.

Automatic visibility evaluation method for application in virtual prototyping environment

ABSTRACTThe evaluation of visibility factor is an essential activity in the ergonomic analysis and verification process. It influences the product quality and production efficiency significantly. However, manual evaluation of the visibility factor in a virtual prototyping environment is complicated and time-consuming owing to the high complexity of products and manufacturing processes. This paper presents an automatic visibility evaluation method. A view-region model that conforms to the physiological characteristics of field-of-view (FOV) is designed and constructed. The surfaces of the observed-object are discretised into a series of triangular meshes, which function as the evaluation units. A cluster of radial test lines, which represent the rays reflected from each mesh, are generated. The visible types of these meshes are assessed by detecting and analysing these test lines. The authors develop a set of computation models for evaluating various visibility-influencing factors. In addition, the overall quantitative and qualitative visibility evaluation results are provided. A case study is illustrated and the results reveal that this method can evaluate the visibility effectively. Furthermore, the potential visibility-related ergonomic problems can be detected and addressed in advance with the aid of this method.

Understanding Actors and Evaluating Personae with Gaussian Embeddings

Understanding narrative content has become an increasingly popular topic. Nonetheless, research on identifying common types of narrative characters, or personae, is impeded by the lack of automatic and broad-coverage evaluation methods. We argue that computationally modeling actors provides benefits, including novel evaluation mechanisms for personae. Specifically, we propose two actor-modeling tasks, cast prediction and versatility ranking, which can capture complementary aspects of the relation between actors and the characters they portray. For an actor model, we present a technique for embedding actors, movies, character roles, genres, and descriptive keywords as Gaussian distributions and translation vectors, where the Gaussian variance corresponds to actors’ versatility. Empirical results indicate that (1) the technique considerably outperforms TransE (Bordes et al. 2013) and ablation baselines and (2) automatically identified persona topics (Bamman, O’Connor, and Smith 2013) yield statistically significant improvements in both tasks, whereas simplistic persona descriptors including age and gender perform inconsistently, validating prior research.

Automatic evaluation of stratum basale and dermal papillae using ultrahigh resolution optical coherence tomography

Diagnosis of many skin conditions requires evaluation of dermal papillae and stratum basale, such as vitiligo. In clinical practice, imaging dermal papillae structures relies on excisional biopsy followed by histological processing and analysis. As biopsy is invasive and associated with complications, a noninvasive imaging method such as optical coherence tomography (OCT) can complement the existing method by enabling large area scanning. However, because OCT image analysis requires training and it takes time to review OCT images from large skin areas, an automatic evaluation method is preferred to reduce the workload and avoid ‘sampling errors’ during image analysis. Here we report an automatic method to enhance and detect dermal papillae and stratum basale in ultrahigh resolution OCT images. A high detection accuracy is achieved by rejecting image artifacts using a surface flattening algorithm and an artifact recognition algorithm. We further demonstrated the efficacy of this automatic method in detecting vitiligo in human subjects.

Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency

Abstractive text summarization is a highly difficult problem, and the sequence-to-sequence model has shown success in improving the performance on the task. However, the generated summaries are often inconsistent with the source content in semantics. In such cases, when generating summaries, the model selects semantically unrelated words with respect to the source content as the most probable output. The problem can be attributed to heuristically constructed training data, where summaries can be unrelated to the source content, thus containing semantically unrelated words and spurious word correspondence. In this article, we propose a regularization approach for the sequence-to-sequence model and make use of what the model has learned to regularize the learning objective to alleviate the effect of the problem. In addition, we propose a practical human evaluation method to address the problem that the existing automatic evaluation method does not evaluate the semantic consistency with the source content properly. Experimental results demonstrate the effectiveness of the proposed approach, which outperforms almost all the existing models. Especially, the proposed approach improves the semantic consistency by 4% in terms of human evaluation.

Analysis and evaluation on the quality of news text machine translation based on neural network

Machine translation quality estimation is an important task in natural language processing. Unlike traditional automatic machine translation evaluation methods, the quality of machine translation is evaluated by the translation quality estimation method without using manual reference translation. In view of the fact that the feature extraction of current sentence-level machine translation quality estimation relies heavily on linguistic analysis, which leads to insufficient generalization ability and restricts the performance of subsequent support vector regression algorithm, it is proposed to extract the features of sentence vectors by using contextual word prediction model and matrix decomposition model in deep learning, and combine them with the features of recurrent neural network language model to improve the correlation between automatic estimation of translation quality and manual evaluation. The experimental results on WMT15 and WMT16 translation quality estimation subtask data sets show that the performance statistics of the method of extracting sentence vector features by adopting context word prediction model are consistently superior to that of the traditional Quest method and continuous space language model sentence vector feature extraction method. It reveals that the proposed feature extraction method not only requires no linguistic analysis, but also significantly improves the effect of translation quality estimation.

A visual automatic analysis and evaluation method based on virtual reality in microgravity environment

To ensure the safety of the astronauts and the prolongation of the life of the space station, maintainability design and verification is an integral part of design process. As a crucial factor in maintainability design, visibility evaluation is the most important part of maintainability design and verification. However, completely different ergonomics parameters and lacking of relevant research are the two major problems limiting the development and application of the visibility design in the microgravity environment. This paper proposes a visibility automation analysis and evaluation model in microgravity environment based on virtual maintenance. Firstly, visibility evaluation indicators are determined by the literature research and the astronauts working condition investigations. Secondly, in responds to the indicators, this paper proposes an evaluation method combining ergonomics and maintainability criteria based on microgravity. Moreover, a comprehensive visual accessibility evaluation model in microgravity environment is developed. Finally, a case of ORU disassembly in space station is conducted and verifies the effectiveness of the proposed method.

СПЕЦИФИКА ОЦЕНИВАНИЯ ХАРАКТЕРИСТИК СЛОЖНЫХ ПОМЕХ С УЧЕТОМ МЕЖКАНАЛЬНОЙ КОРРЕЛЯЦИИ НА ГИПЕРСПЕКТРАЛЬНЫХ ИЗОБРАЖЕНИЯХ

The article deals with the scatter-plot method of automatic evaluation of mixed noise characteristics in multi-channel images. The aim is to solve the problematic issues associated with the adaptation of this method to hyperspectral images processing. The tasks to be solved are: to investigate the influence of the formation method of the jointly processed multichannel groups of images on the accuracy and stability of the aforementioned method; to formulate the recommendations on the choice of jointly processed images and the method of combination. The applied methods are the following: robust estimation of signal parameters, spectral, correlation and regression analysis. The following results were obtained. Three possible ways of groups forming of three channel images were considered: 1) joint processing of images belonging to adjacent channels; 2) joint processing of images with the highest cross-correlation coefficients; 3) joint processing of images with the lowest cross-correlation coefficients. It was defined that if the cross-correlation coefficients of images in the group are low, and the images have of complex structure, it is possible a significant reduction of the method accuracy, up to a complete loss of its working capacity. The method demonstrates sufficiently high accuracy and stability when the groups are formed of the neighbor channel images or of the images with the highest cross-correlation coefficients, and the values of the estimated noise parameters for these issues have no significant differences. Conclusions. The group formation method significantly affects not only the accuracy, but also the operability of the considered estimation method, and in order to increase the reliability of the method, it is appropriate to form groups of images with rather high levels of inter-channel correlation. However, since the accuracy of the method when groups are formed of the neighbor images and of images with the highest levels of cross-correlation have no significant differences, in order to maintain the high performance of the method, it is recommended to form groups of jointly processed images applying the images obtained in the neighbor spectral zones

A Real-Time, Automatic MCG Signal Quality Evaluation Method Using the Magnetocardiography and Electrocardiography

The quality of the obtained Magnetocardiogram (MCG) signal has significant influence on the reliability of deriving some important parameters, such as the interval between two R waves (RR interval), the interval between Q wave and the adjacent T wave (QT interval) and the interval between S wave and the adjacent T wave (ST interval), etc. The poor MCG signal might falsely trigger alarms frequently. The induced overloading of alarms can lead to alarm fatigue, and eventually reduce the care staffs’ attention level. Therefore, a real-time MCG signal quality evaluation algorithm is of great importance. It can allow inexperienced care staffs to collect MCG signals with sufficiently good diagnostic quality. In this work, we proposed an automatic MCG signal quality evaluation algorithm based on the multiparameters derived from MCG data and electrocardiogram (ECG) data. These MCG and ECG data are collected simultaneously. Unlike the available methods for evaluating the signal quality, seven parameters were extracted from the MCG and ECG data. For evaluation purpose, each parameter goes into two categories (good or poor) according to the set threshold. The quality of the MCG signal is evaluated according to the majority rule. The performances of the proposed algorithm were tested in a nine-channel MCG system. The achieved accuracy was 91%. Having demonstrated lower complexity and high accuracy, the proposed algorithm is promising for the real-time MCG signal quality evaluation.

Factor-based evaluation for English to Hindi MT outputs

Design and implementation of automatic evaluation methods is an integral part of any scientific research in accelerating the development cycle of the output. This is no less true for automatic machine translation (MT) systems. However, no such global and systematic scheme exists for evaluation of performance of an MT system. The existing evaluation metrics, such as BLEU, METEOR, TER, although used extensively in literature have faced a lot of criticism from users. Moreover, performance of these metrics often varies with the pair of languages under consideration. The above observation is no less pertinent with respect to translations involving languages of the Indian subcontinent. This study aims at developing an evaluation metric for English to Hindi MT outputs. As a part of this process, a set of probable errors have been identified manually as well as automatically. Linear regression has been used for computing weight/penalty for each error, while taking human evaluations into consideration. A sentence score is computed as the weighted sum of the errors. A set of 126 models has been built using different single classifiers and ensemble of classifiers in order to find the most suitable model for allocating appropriate weight/penalty for each error. The outputs of the models have been compared with the state-of-the-art evaluation metrics. The models developed for manually identified errors correlate well with manual evaluation scores, whereas the models for the automatically identified errors have low correlation with the manual scores. This indicates the need for further improvement and development of sophisticated linguistic tools for automatic identification and extraction of errors. Although many automatic machine translation tools are being developed for many different language pairs, there is no such generalized scheme that would lead to designing meaningful metrics for their evaluation. The proposed scheme should help in developing such metrics for different language pairs in the coming days.

Development of machine translation technology for assisting health communication: A systematic review.

To (1) characterize how machine translation (MT) is being developed to overcome language barriers in health settings; and (2) based on evaluations presented in the literature, determine which MT approaches show evidence of promise and what steps need to be taken to encourage adoption of MT technologies in health settings. We performed a systematic literature search covering 2006-2016 in major health, engineering, and computer science databases. After removing duplicates, two levels of screening identified 27 articles for full text review and analysis. Our review and qualitative analysis covered application setting, target users, underlying technology, whether MT was used in isolation or in combination with human editing, languages tested, evaluation methods, findings, and identified gaps. Of 27 studies, a majority focused on MT systems for use in clinical settings (n = 18), and eight of these involved speech-based MT systems for facilitating patient-provider communications. Text-based MT systems (n = 19) aimed at generating a range of multilingual health materials. Almost a third of all studies (n = 8) pointed to MT's potential as a starting point before human input. Studies employed a variety of human and automatic MT evaluation methods. In comparison studies, statistical machine translation (SMT) systems were more accurate than rule-based systems when large corpora were available. For a variety of systems, performance was best for translations of simple, less technical sentences and from English to Western European languages. Only one system has been fully deployed. MT is currently being developed primarily through pilot studies to improve multilingual communication in health settings and to increase access to health resources for a variety of languages. However, continued concerns about accuracy limit the deployment of MT systems in these settings. The variety of piloted systems and the lack of shared evaluation criteria will likely continue to impede adoption in health settings, where excellent accuracy and a strong evidence base are critical. Greater translation accuracy and use of standard evaluation criteria would encourage deployment of MT into health settings. For now, the literature points to using MT in health communication as an initial step to be followed by human correction.

Maximum Entropy modeling for habitat suitability assessment of Red-crowned crane

Habitat suitability assessment is an important approach for conserving and restoring biodiversity and waterfowl’s habitats. Previous habitat suitability index (HSI) models define the weights of each habitat feature subjectively in their approach. To address this problem, we utilized a data-driven Maximum Entropy (MaxEnt) model to assess the fine-scaled habitat suitability of Red-crowned crane (Grus japonensis) in the breeding period based on the occurrence locations of Red-crowned crane and the habitat features derived from optical, radar imagery and topographical ancillary data. Results of this research show that the developed MaxEnt model improves the performances of previous HSI models. The MaxEnt model could identify the influences of the selected habitat features on the habitat suitability of the species and quantify the optimal habitat conditions for the Red-crowned cranes automatically. We found that habitat composition, water depth, and distance to roads and ditches were most important habitat features for Red-crowned cranes in the breeding period. The optimal conditions for each selected habitat features were recognized according to the response curves of environmental variables to occurrence probability of the species. For the breeding Red-crowned cranes, the desirable land cover type of breeding habitat is reed swamp; the suitable depth of water under the canopy is between 15 and 30 cm approximately; the selected vegetation cover is about 47.5%–74.5%; and the suitable distance from human disturbances is above 2500 m. This study demonstrated the practicability of the developed species distribution model (MaxEnt) on habitat suitability assessment for Red-crowned cranes, and provided a quantitative and automatic habitat suitability evaluation method for endangered waterfowl’s protection.

Automatic evaluation of traumatic brain injury based on terahertz imaging with machine learning.

The imaging diagnosis and prognostication of different degrees of traumatic brain injury (TBI) is very important for early care and clinical treatment. Especially, the exact recognition of mild TBI is the bottleneck for current label-free imaging technologies in neurosurgery. Here, we report an automatic evaluation method for TBI recognition with terahertz (THz) continuous-wave (CW) transmission imaging based on machine learning (ML). We propose a new feature extraction method for biological THz images combined with the transmittance distribution features in spatial domain and statistical distribution features in normalized gray histogram. Based on the extracted feature database, ML algorithms are performed for the classification of different degrees of TBI by feature selection and parameter optimization. The highest classification accuracy is up to 87.5%. The area under the curve (AUC) scores of the receiver operating characteristics (ROC) curve are all higher than 0.9, which shows this evaluation method has a good generalization ability. Furthermore, the excellent performance of the proposed system in the recognition of mild TBI is analyzed by different methodological parameters and diagnostic criteria. The system can be extensible to various diseases and will be a powerful tool in automatic biomedical diagnostics.

Pavement Crack Detection Using Sketch Token

Due to the serious threats to road safety caused by pavement distress, it has drawn much attention to seek efficient road crack detection methods for automatic pavement condition evaluation. In recent detectors, structure information of cracks has not been utilized effectively, which leads to the high sensitivity to crack-like noises. In this paper, we propose a novel crack detection method to recognize cracks out of noises. We verified our method in a real pavement crack dataset. Extensive experiments demonstrate the state-of-the-art results of the proposed method.

A Safety Evaluation-based Image Quality Method for Roads and Bridges

A computer-aided automatic safety evaluation method is proposed based on quality evaluation on digital images of roads or bridges and other image information collected by highway monitoring devices. Images of qualified roads or bridges are selected to form a reference image database, and reference image sequence and evaluation image sequence are established separately. Then combined with the peak signal to noise ratio (PSNR) and the human visual characteristic information entropy, a safety evaluation function with dynamic weights is obtained. At last, the evaluating algorithm is used to compare similarities between evaluation images and reference images to judge the quality of roads or bridges and get a sequence of evaluation parameters sequence. If the value of the evaluation parameter is greater than the threshold, the road or bridge quality changes greatly, and therefore artificial inspection is required. The experimental results show that the evaluation is consistent with the subjective perception of human vision, and the method proposed in this paper has high degree of automation.

A User Interface Design Method based on Quantitative Evaluation of Consistency in Device Operation

SUMMARYDigitized electrical products have many functions, and these user interface specifications have become large scale. Therefore, in short development period, it is a problem that time for usability evaluation is insufficient. In this paper, we propose a user interface design method that can evaluate consistency in device operation, by simple processes. Features of the proposed method are to use similarity of functions or interactions, as software metrics. The similarities are calculated from the design information, and are analyzed by using an agglomerative hierarchical clustering (AHC). As a result of the AHC, functions or interactions are classified into several groups, and their consistencies are able to be judged. This is an automatic evaluation method that does not require manpower. Thus, unlike conventional usability evaluation, efficiency of the design work can be expected. As a result of evaluating user interfaces for digital camera by using the method, this technique was able to identify the points to be corrected in the interactions.

A new retrieval method based on time series variation using field association terms

This paper studies the effects of the time change on the frequencies of specific terms connected to the document field in a given period. These specific terms are the field association (FA) terms. The paper also suggests a new method for automatic evaluation of the stabilization classes of FA terms to improve the precision of decision tree. The stabilization classes point out the popularity of list of FA terms depending on time change. Moreover, the suggested method manipulates the problem of the scattering of data numbers among classes to improve the performance of decision tree precision. The presented method is evaluated through conducting experiments by simulating the result of 1245 files, which are equivalent to 4.15 MB. The F‐measure for increment, fairly steady, and decrement classes achieves %90.4, %99.3, and %38.6, sequentially.

Automated evaluation of agricultural damage using UAV survey

Abstract In the last decade modern agriculture used significant resources for automated evaluation. Game damage and crop evaluation and prediction are the main focus of the precision agriculture. At John von Neumann Faculty of Informatics, Óbuda University, Hungary, researchers together with agricultural experts are developing automated evaluation methods for high-resolution survey data. With the locally developed unmanned aerial vehicles (UAVs) the agricultural survey can be easily done for 1-2km2 area from a single flight. The paper presents the detailed method of the early steps of the automated evaluation using Matlab and the comparison of results to the GIS data.