Automatic Quality Assessment Research Articles

Algorithmic assessment of drag on thermally cut sheet metal edges

Abstract Drag is a key criterion in assessing the quality of thermally cut sheet metal edges, which is critical to the reliability of the final product. The evaluation of drag has been described qualitatively and quantitatively, but the scientific literature lacks a methodical description of algorithmic tracking of the drag lines themselves. This absence of a standardized approach has hindered the objective determination of drag. With recent advances in the field towards automated quality assessment aimed at autonomous adaptation of process parameters, the need for consistent, fast and reliable assessment of drag lines has become apparent. To address this gap, this study introduces an innovative drag line tracking algorithm, inspired by the behavior of fluid flowing towards the lowest points, to compute a generalized drag line for an edge with a homogeneous cutting pattern. The algorithm utilizes the height data of the measured cut edges as a data base for the assessment of the drag lines. The results indicate that the drag lines identified by the algorithm are not only subjectively accurate, but also show a strong correlation with human-annotated drag lines across several metrics. This work lays the foundation for the objective evaluation of drag by not only describing an algorithm for the consistent determination of drag lines, but also by presenting a tool for human annotation and suitable customized metrics. As a result, it contributes significantly to the comprehensive evaluation of edge quality and represents a step forward in the automatic optimization of process parameters and the improvement of cutting edge quality.

Automatic ploidy prediction and quality assessment of human blastocysts using time-lapse imaging

Assessing fertilized human embryos is crucial for in vitro fertilization, a task being revolutionized by artificial intelligence. Existing models used for embryo quality assessment and ploidy detection could be significantly improved by effectively utilizing time-lapse imaging to identify critical developmental time points for maximizing prediction accuracy. Addressing this, we develop and compare various embryo ploidy status prediction models across distinct embryo development stages. We present BELA, a state-of-the-art ploidy prediction model that surpasses previous image- and video-based models without necessitating input from embryologists. BELA uses multitask learning to predict quality scores that are thereafter used to predict ploidy status. By achieving an area under the receiver operating characteristic curve of 0.76 for discriminating between euploidy and aneuploidy embryos on the Weill Cornell dataset, BELA matches the performance of models trained on embryologists’ manual scores. While not a replacement for preimplantation genetic testing for aneuploidy, BELA exemplifies how such models can streamline the embryo evaluation process.

Quality Assessment of Brain MRI Defacing Using Machine Learning.

Defacing of brain magnetic resonance imaging (MRI) scans is a crucial process in medical imaging research aimed at preserving patient privacy while maintaining data integrity. However, existing defacing algorithms are prone to errors, potentially compromising patient anonymity. This paper investigates the feasibility and efficacy of automated quality assessment for defaced brain MRIs using machine learning (ML). Our findings demonstrate the promising capability of ML models in accurately distinguishing between properly and inadequately defaced MRI scans.

Interactive Cascaded Network for Prostate Cancer Segmentation from Multimodality MRI with Automated Quality Assessment.

The accurate segmentation of prostate cancer (PCa) from multiparametric MRI is crucial in clinical practice for guiding biopsy and treatment planning. Existing automated methods often lack the necessary accuracy and robustness in localizing PCa, whereas interactive segmentation methods, although more accurate, require user intervention on each input image, thereby limiting the cost-effectiveness of the segmentation workflow. Our innovative framework addresses the limitations of current methods by combining a coarse segmentation network, a rejection network, and an interactive deep network known as Segment Anything Model (SAM). The coarse segmentation network automatically generates initial segmentation results, which are evaluated by the rejection network to estimate their quality. Low-quality results are flagged for user interaction, with the user providing a region of interest (ROI) enclosing the lesions, whereas for high-quality results, ROIs were cropped from the automatic segmentation. Both manually and automatically defined ROIs are fed into SAM to produce the final fine segmentation. This approach significantly reduces the annotation burden and achieves substantial improvements by flagging approximately 20% of the images with the lowest quality scores for manual annotation. With only half of the images manually annotated, the final segmentation accuracy is statistically indistinguishable from that achieved using full manual annotation. Although this paper focuses on prostate lesion segmentation from multimodality MRI, the framework can be adapted to other medical image segmentation applications to improve segmentation efficiency while maintaining high accuracy standards.

Automatic Quality Assessment of Pork Belly via Deep Learning and Ultrasound Imaging.

Pork belly, prized for its unique flavor and texture, is often overlooked in breeding programs that prioritize lean meat production. The quality of pork belly is determined by the number and distribution of muscle and fat layers. This study aimed to assess the number of pork belly layers using deep learning techniques. Initially, semantic segmentation was considered, but the intersection over union (IoU) scores for the segmented parts were below 70%, which is insufficient for practical application. Consequently, the focus shifted to image classification methods. Based on the number of fat and muscle layers, a dataset was categorized into three groups: three layers (n = 1811), five layers (n = 1294), and seven layers (n = 879). Drawing upon established model architectures, the initial model was refined for the task of learning and predicting layer traits from B-ultrasound images of pork belly. After a thorough evaluation of various performance metrics, the ResNet18 model emerged as the most effective, achieving a remarkable training set accuracy of 99.99% and a validation set accuracy of 96.22%, with corresponding loss values of 0.1478 and 0.1976. The robustness of the model was confirmed through three interpretable analysis methods, including grad-CAM, ensuring its reliability. Furthermore, the model was successfully deployed in a local setting to process B-ultrasound video frames in real time, consistently identifying the pork belly layer count with a confidence level exceeding 70%. By employing a scoring system with 100 points as the threshold, the number of pork belly layers in vivo was categorized into superior and inferior grades. This innovative system offers immediate decision-making support for breeding determinations and presents a highly efficient and precise method for assessment of pork belly layers.

Automatic quality assessment of knee radiographs using knowledge graphs and convolutional neural networks.

X-ray radiography is a widely used imaging technique worldwide, and its image quality directly affects diagnostic accuracy. Therefore, X-ray image quality control (QC) is essential. However, subjectively assessing image quality is inefficient and inconsistent, especially when large amounts of image data are being evaluated. Thus, subjective assessment cannot meet current QC needs. To meet current QC needs and improve the efficiency of image quality assessment, a complete set of quality assessment criteria must be established and implemented using artificial intelligence (AI) technology. Therefore, we proposed a multi-criteria AI system for automatically assessing the image quality of knee radiographs. A knee radiograph QC knowledge graph containing 16 "acquisition technique" labels representing 16 image quality defects and five "clarity" labels representing five grades of clarity were developed. Ten radiographic technologists conducted three rounds of QC based on this graph. The single-person QC results were denoted as QC1 and QC2, and the multi-person QC results were denoted as QC3. Each technologist labeled each image only once. The ResNet model structure was then used to simultaneously perform classification (detection of image quality defects) and regression (output of a clarity score) tasks to construct an image QC system. The QC3 results, comprising 4324 anteroposterior and lateral knee radiographs, were used for model training (70% of the images), validation (10%), and testing (20%). The 865 test set data were used to evaluate the effectiveness of the AI model, and an AI QC result, QC4, was automatically generated by the model after training. Finally, using a double-blind method, the senior QC expert reviewed the final QC results of the test set with reference to the results QC3 and QC4 and used them as a reference standard to evaluate the performance of the model. The precision and mean absolute error (MAE) were used to evaluate the quality of all the labels in relation to the reference standard. For the 16 "acquisition technique" features, QC4 exhibited the highest weighted average precision (98.42%±0.81%), followed by QC3 (91.39%±1.35%), QC2 (87.84%±1.68%), and QC1 (87.35%±1.71%). For the image clarity features, the MAEs between QC1, QC2, QC3, and QC4 and the reference standard were 0.508±0.021, 0.475±0.019, 0.237±0.016, and 0.303±0.018, respectively. The experimental results show that our automated quality assessment system performed well in classifying the acquisition technique used for knee radiographs. The image clarity quality evaluation accuracy of the model must be further improved but is generally close to that of radiographic technologists. Intelligent QC methods using knowledge graphs and convolutional neural networks have the potential for clinical applications.

How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?

Software metrics such as coverage or mutation scores have been investigated for the automated quality assessment of test suites. While traditional tools rely on software metrics, the field of self-driving cars (SDCs) has primarily focused on simulation-based test case generation using quality metrics such as the out-of-bound (OOB) parameter to determine if a test case fails or passes. However, it remains unclear to what extent this quality metric aligns with the human perception of the safety and realism of SDCs. To address this (reality) gap, we conducted an empirical study involving 50 participants to investigate the factors that determine how humans perceive SDC test cases as safe, unsafe, realistic, or unrealistic. To this aim, we developed a framework leveraging virtual reality (VR) technologies, called SDC-Alabaster, to immerse the study participants into the virtual environment of SDC simulators. Our findings indicate that the human assessment of safety and realism of failing/passing test cases can vary based on different factors, such as the test's complexity and the possibility of interacting with the SDC. Especially for the assessment of realism, the participants' age leads to a different perception. This study highlights the need for more research on simulation testing quality metrics and the importance of human perception in evaluating SDCs.

Automated Quality Assessment of Medical Images in Echocardiography Using Neural Networks with Adaptive Ranking and Structure-Aware Learning.

The quality of medical images is crucial for accurately diagnosing and treating various diseases. However, current automated methods for assessing image quality are based on neural networks, which often focus solely on pixel distortion and overlook the significance of complex structures within the images. This study introduces a novel neural network model designed explicitly for automated image quality assessment that addresses pixel and semantic distortion. The model introduces an adaptive ranking mechanism enhanced with contrast sensitivity weighting to refine the detection of minor variances in similar images for pixel distortion assessment. More significantly, the model integrates a structure-aware learning module employing graph neural networks. This module is adept at deciphering the intricate relationships between an image's semantic structure and quality. When evaluated on two ultrasound imaging datasets, the proposed method outshines existing leading models in performance. Additionally, it boasts seamless integration into clinical workflows, enabling real-time image quality assessment, crucial for precise disease diagnosis and treatment.

Implementation and evaluation of a system for assessment of the quality of long-term management of patients at a geriatric hospital

BackgroundThe increasing aging population presents a significant challenge, accompanied by a shortage of professional caregivers, adding to the therapeutic burden. Clinical decision support systems, utilizing computerized clinical guidelines, can improve healthcare quality, reduce expenses, save time, and boost caregiver efficiency. Objectives1) Develop and evaluate an automated quality assessment (QA) system for retrospective longitudinal care quality analysis, focusing on clinical staff adherence to evidence-based guidelines (GLs). 2) Assess the system’s technical feasibility and functional capability for senior nurse use in geriatric pressure-ulcer management. MethodsA computational QA system using our Quality Assessment Temporal Patterns (QATP) methodology was designed and implemented. Our methodology transforms the GL’s procedural-knowledge into declarative-knowledge temporal-abstraction patterns representing the expected execution trace in the patient’s data for correct therapy application. Fuzzy temporal logic allows for partial compliance, reflecting individual and grouped action performance considering their values and temporal aspects. The system was tested using a pressure ulcer treatment GL and data from 100 geriatric patients’ Electronic Medical Records (EMR). After technical evaluation for accuracy and feasibility, an extensive functional evaluation was conducted by an experienced nurse, comparing QA scores with and without system support, and versus automated system scores. Time efficiency was also measured. ResultsQA scores from the geriatric nurse, with and without system’s support, did not significantly differ from those provided by the automated system (p < 0.05), demonstrating the effectiveness and reliability of both manual and automated methods. The system-supported manual QA process reduced scoring time by approximately two-thirds, from an average of 17.3 min per patient manually to about 5.9 min with the system’s assistance, highlighting the system’s efficiency potential in clinical practice. ConclusionThe QA system based on QATP, produces scores consistent with an experienced nurse’s assessment for complex care over extended periods. It enables quick and accurate quality care evaluation for multiple patients after brief training. Such automated QA systems may empower nursing staff, enabling them to manage more patients, accurately and consistently, while reducing costs due to saved time and effort, and enhanced compliance with evidence-based guidelines.

Towards assessing the quality of knowledge graphs via differential testing

Knowledge graphs (KG) can aggregate data and make information resources easier to calculate and understand. With tremendous advancements in knowledge graphs, they have been incorporated into plenty of software systems to assist various tasks. However, while KGs determine the performance of downstream software systems, their quality is often measured by the accuracy of test data. Considering the limitation of accessible high-quality test data, an automated quality assessment technique could fundamentally improve the testing efficiency of KG-driven software systems and save plenty of manual labeling resources.In this paper, we propose an automated approach to quantify the quality of KGs via differential testing. It first constructs multiple Knowledge Graph Embedding Models (KGEM) and conducts head prediction tasks on models. Then, it can produce a differential score that reflects the quality of KGs by comparing the proximity of output results. To validate the effectiveness of this approach, we experiment with four open-sourced knowledge graphs. The experiment results show that our approach is capable of accurately evaluating the quality of KGs and producing reliable results on different datasets. Moreover, we compared our method with existing methods and achieved certain advantages. The potential usefulness of our approach sheds light on the development of various KG-driven software systems.

What patient positioning in chest X-ray is still acceptable? – An empirical study on thresholding quality metrics

AbstractBackground and AimIssues in patient positioning during chest X-ray (CXR) acquisition impair diagnostic quality and potentially increase radiation dose. Automated quality assessment was proposed to address this. Our objective is to determine thresholds on some quality control metrics following international guidelines, that represent expert knowledge and can be applied in a comprehensible and explainable AI approach for such an automatic quality assessment.Materials and MethodsAn AI-method estimating collimation distance to the ribcage, balancing between both clavicle heads, and number of ribs above the diaphragm as metrics for collimation, rotation, and inhalation quality was applied on 64,315 posteroanterior CXR images from a public dataset (ChestX-ray8). From this set 920 CXR images were sampled and manually annotated to gain additional trusted reference metrics. Seven readers from different institutions then classified the acquisition quality of these images independently into okay, inadequate, or unacceptable following the criteria of international guidelines. Optimal thresholds on the metrics were determined to reproduce these classes using the metrics only.ResultsA fair to moderate agreement between the experts was found. When disregarding all inadequate rates a classification on the metrics was able to separate okay rated cases from unacceptable cases for collimation (AUC 0.97), rotation (AUC = 0.93) and inhalation (AUC = 0.97).ConclusionSuitable thresholds were determined to reproduce expert opinions in the assessment of the most important quality criteria in CXR acquisition. These thresholds were finally applied on the AI-method's estimates to automatically classify image acquisition quality comprehensibly and according to the guidelines.

Multiple Image Distortion DNN Modeling Individual Subject Quality Assessment

A recent research direction is focused on training Deep Neural Networks (DNNs) to replicate individual subject assessments of media quality. These DNNs are referred to as Artificial Intelligence-based Observers (AIOs). An AIO is designed to simulate, in real-time, the quality ratings of a specific individual, enabling an automatic quality assessment that accounts for subjects characteristics and preferences. Training AIOs is a promising but challenging research area due to the greater noise in individual raw opinion scores compared to the Mean Opinion Score (MOS). Effective learning from noisy labels necessitates the training of complex models on large-scale datasets. Unfortunately, this is challenging for AIOs as the media quality assessment community lacks extensive datasets that include individual opinion scores. To address the complexity of the task, we first created a dataset comprising two million samples, with synthetic labels derived from human annotation. We then trained a customized network for image quality assessment, named Multi-Distortion ResNet50 (MDResNet50), on this dataset. The weights of the MDResNet50 were subsequently utilized to initialize the learning process of each AIO, thereby avoiding the need to train a complex model from scratch on a small-scale dataset with raw individual opinion scores. Computational experiments show that our approach significantly advances the state-of-the-art in the AIO research. In particular: i) we demonstrate through a simulation the ability of AIOs to mimic two well-known behavioral characteristics of a subject, i.e.,bias and inconsistency, when scoring the media quality; ii) we train and release DNN-based AIOs that, compared to the state-of-the-art, exhibit a higher performance with a statistical significance in assessing multiple image distortions; iii) we train AIOs that more accurately mimic the sensitivity of real subjects to noise and color saturation, and also better predict the opinion score distribution compared to the state-of-the-art AIOs.

Automatic Resume Quality Assessment

In the ever-evolving and competitive job market, presenting a compelling resume has become critical for job seekers to secure interviews and land their desired positions. However, manually reviewing and assessing a vast number of resumes can be a time- consuming and laborious task for re- cruiters, often leading to inefficiencies and potential biases in the hiring process. Automatic Resume Quality Assessment (ARQA) systems have emerged as promising solutions to address these challenges, leveraging the power of artificial intelligence (AI) and natural language processing (NLP) techniques to automate the resume evaluation process. This survey paper delves into the fascinating world of ARQA, providing a comprehensive overview of the existing approaches, techniques, challenges, and promisingfuture directions.

Deep learning-driven automated quality assessment of ultra-widefield optical coherence tomography angiography images for diabetic retinopathy

AbstractImage quality assessment (IQA) of fundus images constitutes a foundational step in automated disease analysis. This process is pivotal in supporting the automation of screening, diagnosis, follow-up, and related academic research for diabetic retinopathy (DR). This study introduced a deep learning-based approach for IQA of ultra-widefield optical coherence tomography angiography (UW-OCTA) images of patients with DR. Given the novelty of ultra-widefield technology, its limited prevalence, the high costs associated with equipment and operational training, and concerns regarding ethics and patient privacy, UW-OCTA datasets are notably scarce. To address this, we initially pre-train a vision transformer (ViT) model on a dataset comprising 6 mm × 6 mm OCTA images, enabling the model to acquire a fundamental understanding of OCTA image characteristics and quality indicators. Subsequent fine-tuning on 12 mm × 12 mm UW-OCTA images aims to enhance accuracy in quality assessment. This transfer learning strategy leverages the generic features learned during pre-training and adjusts the model to evaluate UW-OCTA image quality effectively. Experimental results demonstrate that our proposed method achieves superior performance compared to ResNet18, ResNet34, and ResNet50, with an AUC of 0.9026 and a Kappa value of 0.7310. Additionally, ablation studies, including the omission of pre-training on 6 mm × 6 mm OCTA images and the substitution of the backbone network with the ViT base version, resulted in varying degrees of decline in AUC and Kappa values, confirming the efficacy of each module within our methodology.

Deep learning-assisted automatic quality assessment of concrete surfaces with cracks and bugholes

Cracks and bugholes are common factors affecting the quality of concrete surface, thus automatic detection of apparent defects is essential for concrete surface quality assessment. This study proposes an automatic quality assessment method based on deep learning for concrete surfaces with different scales of cracks and bugholes. In particular, a modified cascade mask region-based convolutional neural network (MC Mask R-CNN) with ResNet and feature pyramid network (FPN) is presented. Furthermore, a novel quantitative analysis algorithm for instance segmentation and size quantification of cracks and bugholes is proposed. Multi-scale features extracted by ResNet can be integrated using FPN. Closed operations and fixed elliptical kernels are adopted to reduce inaccuracies associated with crack discontinuities. Datasets collected at construction sites are used for training and testing of the network after data augmentation. Comparison work was carried out to evaluate the proposed model, and the box mean average precision (APbbox) and the mask mean average precision (APmask) index can reach up to 0.588 and 0.553. The overall quantization errors of cracks and bugholes were within acceptable limits. Validation experiments were conducted to illustrate the validity of the quality assessment method.

A deep learning-based automatic image quality assessment method for respiratory phase on computed tomography chest images.

Computed tomography (CT) chest scans have become commonly used in clinical diagnosis. Image quality assessment (IQA) for CT images plays an important role in CT examination. It is worth noting that IQA is still a manual and subjective process, and even experienced radiologists make mistakes due to human limitations (fatigue, perceptual biases, and cognitive biases). There are also kinds of biases because of poor consensus among radiologists. Excellent IQA methods can reliably give an objective evaluation result and also reduce the workload of radiologists. This study proposes a deep learning (DL)-based automatic IQA method, to assess whether the image quality of respiratory phase on CT chest images are optimal or not, so that the CT chest images can be used in the patient's physical condition assessment. This retrospective study analysed 212 patients' chest CT images, with 188 patients allocated to a training set (150 patients), validation set (18 patients), and a test set (20 patients). The remaining 24 patients were used for the observer study. Data augmentation methods were applied to address the problem of insufficient data. The DL-based IQA method combines image selection, tracheal carina segmentation, and bronchial beam detection. To automatically select the CT image containing the tracheal carina, an image selection model was employed. Afterward, the area-based approach and score-based approach were proposed and used to further optimize the tracheal carina segmentation and bronchial beam detection results, respectively. Finally, the score about the image quality of the patient's respiratory phase images given by the DL-based automatic IQA method was compared with the mean opinion score (MOS) given in the observer study, in which four blinded experienced radiologists took part. The DL-based automatic IQA method achieved good performance in assessing the image quality of the respiratory phase images. For the CT sequence of the same patient, the DL-based IQA method had an accuracy of 92% in the assessment score, while the radiologists had an accuracy of 88%. The Kappa value of the assessment score between the DL-based IQA method and radiologists was 0.75, with a sensitivity of 85%, specificity of 91%, positive predictive value (PPV) of 92%, negative predictive value (NPV) of 93%, and accuracy of 88%. This study develops and validates a DL-based automatic IQA method for the respiratory phase on CT chest images. The performance of this method surpassed that of the experienced radiologists on the independent test set used in this study. In clinical practice, it is possible to reduce the workload of radiologists and minimize errors caused by human limitations.

Advancing Emergency Supplies Management: A Blockchain‐Based Traceability System for Cold‐Chain Medicine Logistics

AbstractAdvancing cold‐chain medicine logistics and emergency supplies management necessitates a robust blockchain‐based traceability system. This specialized information system is essential for enhancing quality control and operational efficiency. Despite challenges like insufficient supplies, ineffective distribution, and delays, blockchain's transparency and traceability offer significant potential to revolutionize national emergency management. This paper presents a blockchain‐based traceability system tailored for cold‐chain medicine logistics. This system achieves traceability and automatic medicine quality assessment by integrating blockchain principles and leveraging hazard analysis theory combined with IoT technology. This system also enhances data reliability with a multi‐sensor data fusion algorithm, improving temperature accuracy beyond conventional methods. Additionally, the feasibility and necessity of integrating blockchain into cold‐chain pharmaceutical traceability processes is thoroughly examined. The proposed system fulfills traceability needs for emergency cold‐chain medicine while addressing industry challenges.

Deep Learning Identifies High-Quality Fundus Photographs and Increases Accuracy in Automated Primary Open Angle Glaucoma Detection.

To develop and evaluate a deep learning (DL) model to assess fundus photograph quality, and quantitatively measure its impact on automated POAG detection in independent study populations. Image quality ground truth was determined by manual review of 2815 fundus photographs of healthy and POAG eyes from the Diagnostic Innovations in Glaucoma Study and African Descent and Glaucoma Evaluation Study (DIGS/ADAGES), as well as 11,350 from the Ocular Hypertension Treatment Study (OHTS). Human experts assessed a photograph as high quality if of sufficient quality to determine POAG status and poor quality if not. A DL quality model was trained on photographs from DIGS/ADAGES and tested on OHTS. The effect of DL quality assessment on DL POAG detection was measured using area under the receiver operating characteristic (AUROC). The DL quality model yielded an AUROC of 0.97 for differentiating between high- and low-quality photographs; qualitative human review affirmed high model performance. Diagnostic accuracy of the DL POAG model was significantly greater (P < 0.001) in good (AUROC, 0.87; 95% CI, 0.80-0.92) compared with poor quality photographs (AUROC, 0.77; 95% CI, 0.67-0.88). The DL quality model was able to accurately assess fundus photograph quality. Using automated quality assessment to filter out low-quality photographs increased the accuracy of a DL POAG detection model. Incorporating DL quality assessment into automated review of fundus photographs can help to decrease the burden of manual review and improve accuracy for automated DL POAG detection.

A clinical consensus-compliant deep learning approach to quantitatively evaluate human in vitro fertilization early embryonic development with optical microscope images

The selection of embryos is a key for the success of in vitro fertilization (IVF). However, automatic quality assessment on human IVF embryos with optical microscope images is still challenging. In this study, we developed a clinical consensus-compliant deep learning approach, named Esava (Embryo Segmentation and Viability Assessment), to quantitatively evaluate the development of IVF embryos using optical microscope images. In total 551 optical microscope images of human IVF embryos of day-2 to day-3 were collected, preprocessed, and annotated. Using the Faster R-CNN model as baseline, our Esava model was constructed, refined, trained, and validated for precise and robust blastomere detection. A novel algorithm Crowd-NMS was proposed and employed in Esava to enhance the object detection and to precisely quantify the embryonic cells and their size uniformity. Additionally, an innovative GrabCut-based unsupervised module was integrated for the segmentation of blastomeres and embryos. Independently tested on 94 embryo images for blastomere detection, Esava obtained the high rates of 0.9940, 0.9121, and 0.9531 for precision, recall, and mAP respectively, and gained significant advances compared with previous computational methods. Intraclass correlation coefficients indicated the consistency between Esava and three experienced embryologists. Another test on 51 extra images demonstrated that Esava surpassed other tools significantly, achieving the highest average precision 0.9025. Moreover, it also accurately identified the borders of blastomeres with mIoU over 0.88 on the independent testing dataset. Esava is compliant with the Istanbul clinical consensus and compatible to senior embryologists. Taken together, Esava improves the accuracy and efficiency of embryonic development assessment with optical microscope images.

How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scale

ObjectivesThe aim of this study is to improve the reliability of subjective IQ assessment using a pairwise comparison (PC) method instead of a Likert scale method in abdominal CT scans.MethodsAbdominal CT scans (single-center) were retrospectively selected between September 2019 and February 2020 in a prior study. Sample variance in IQ was obtained by adding artificial noise using dedicated reconstruction software, including reconstructions with filtered backprojection and varying iterative reconstruction strengths. Two datasets (each n = 50) were composed with either higher or lower IQ variation with the 25 original scans being part of both datasets. Using in-house developed software, six observers (five radiologists, one resident) rated both datasets via both the PC method (forcing observers to choose preferred scans out of pairs of scans resulting in a ranking) and a 5-point Likert scale. The PC method was optimized using a sorting algorithm to minimize necessary comparisons. The inter- and intraobserver agreements were assessed for both methods with the intraclass correlation coefficient (ICC).ResultsTwenty-five patients (mean age 61 years ± 15.5; 56% men) were evaluated. The ICC for interobserver agreement for the high-variation dataset increased from 0.665 (95%CI 0.396–0.814) to 0.785 (95%CI 0.676–0.867) when the PC method was used instead of a Likert scale. For the low-variation dataset, the ICC increased from 0.276 (95%CI 0.034–0.500) to 0.562 (95%CI 0.337–0.729). Intraobserver agreement increased for four out of six observers.ConclusionThe PC method is more reliable for subjective IQ assessment indicated by improved inter- and intraobserver agreement.Clinical relevance statementThis study shows that the pairwise comparison method is a more reliable method for subjective image quality assessment. Improved reliability is of key importance for optimization studies, validation of automatic image quality assessment algorithms, and training of AI algorithms.Key Points• Subjective assessment of diagnostic image quality via Likert scale has limited reliability.• A pairwise comparison method improves the inter- and intraobserver agreement.• The pairwise comparison method is more reliable for CT optimization studies.Graphical