Independent Human Observers Research Articles

An open-source tool for automated human-level circling behavior detection

Quantitatively relating behavior to underlying biology is crucial in life science. Although progress in keypoint tracking tools has reduced barriers to recording postural data, identifying specific behaviors from this data remains challenging. Manual behavior coding is labor-intensive and inconsistent, while automatic methods struggle to explicitly define complex behaviors, even when they seem obvious to the human eye. Here, we demonstrate an effective technique for detecting circling in mice, a form of locomotion characterized by stereotyped spinning. Despite circling's extensive history as a behavioral marker, there currently exists no standard automated detection method. We developed a circling detection technique using simple postprocessing of keypoint data obtained from videos of freely-exploring (Cib2−/−;Cib3−/−) mutant mice, a strain previously found to exhibit circling behavior. Our technique achieves statistical parity with independent human observers in matching occurrence times based on human consensus, and it accurately distinguishes between videos of wild type mice and mutants. Our pipeline provides a convenient, noninvasive, quantitative tool for analyzing circling mouse models without the need for software engineering experience. Additionally, as the concepts underlying our approach are agnostic to the behavior being analyzed, and indeed to the modality of the recorded data, our results support the feasibility of algorithmically detecting specific research-relevant behaviors using readily-interpretable parameters tuned on the basis of human consensus.

Automatic Estimation of Interpersonal Engagement During Naturalistic Conversation Using Dyadic Physiological Measurements.

Physiological responses of two interacting individuals contain a wealth of information about the dyad: for example, the degree of engagement or trust. However, nearly all studies on dyadic physiological responses have targeted group-level analysis: e.g., correlating physiology and engagement in a large sample. Conversely, this paper presents a study where physiological measurements are combined with machine learning algorithms to dynamically estimate the engagement of individual dyads. Sixteen dyads completed 15-min naturalistic conversations and self-reported their engagement on a visual analog scale every 60 s. Four physiological signals (electrocardiography, skin conductance, respiration, skin temperature) were recorded, and both individual physiological features (e.g., each participant’s heart rate) and synchrony features (indicating degree of physiological similarity between two participants) were extracted. Multiple regression algorithms were used to estimate self-reported engagement based on physiological features using either leave-interval-out crossvalidation (training on 14 60-s intervals from a dyad and testing on the 15th interval from the same dyad) or leave-dyad-out crossvalidation (training on 15 dyads and testing on the 16th). In leave-interval-out crossvalidation, the regression algorithms achieved accuracy similar to a ‘baseline’ estimator that simply took the median engagement of the other 14 intervals. In leave-dyad-out crossvalidation, machine learning achieved a slightly higher accuracy than the baseline estimator and higher accuracy than an independent human observer. Secondary analyses showed that removing synchrony features and personality characteristics from the input dataset negatively impacted estimation accuracy and that engagement estimation error was correlated with personality traits. Results demonstrate the feasibility of dynamically estimating interpersonal engagement during naturalistic conversation using physiological measurements, which has potential applications in both conversation monitoring and conversation enhancement. However, as many of our estimation errors are difficult to contextualize, further work is needed to determine acceptable estimation accuracies.

R-JaunLab: Automatic Multi-Class Recognition of Jaundice on Photos of Subjects with Region Annotation Networks.

Jaundice occurs as a symptom of various diseases, such as hepatitis, the liver cancer, gallbladder or pancreas. Therefore, clinical measurement with special equipment is a common method that is used to identify the total serum bilirubin level in patients. Fully automated multi-class recognition of jaundice combines two key issues: (1) the critical difficulties in multi-class recognition of jaundice approaches contrasting with the binary class and (2) the subtle difficulties in multi-class recognition of jaundice represent extensive individuals variability of high-resolution photos of subjects, huge coherency between healthy controls and occult jaundice, as well as broadly inhomogeneous color distribution. We introduce a novel approach for multi-class recognition of jaundice to detect occult jaundice, obvious jaundice and healthy controls. First, region annotation network is developed and trained to propose eye candidates. Subsequently, an efficient jaundice recognizer is proposed to learn similarities, context, localization features and globalization characteristics on photos of subjects. Finally, both networks are unified by using shared convolutional layer. Evaluation of the structured model in a comparative study resulted in a significant performance boost (categorical accuracy for mean 91.38%) over the independent human observer. Our work was exceeded against the state-of-the-art convolutional neural network (96.85% and 90.06% for training and validation subset, respectively) and showed a remarkable categorical result for mean 95.33% on testing subset. The proposed network makes a performance better than physicians. This work demonstrates the strength of our proposal to help bringing an efficient tool for multi-class recognition of jaundice into clinical practice.

A generative adversarial network-based (GAN-based) architecture for automatic fiducial marker detection in prostate MRI-only radiotherapy simulation images.

Clinical sites utilizing magnetic resonance imaging (MRI)-only simulation for prostate radiotherapy planning typically use fiducial markers for pretreatment patient positioning and alignment. Fiducial markers appear as small signal voids in MRI images and are often difficult to discern. Existing clinical methods for fiducial marker localization require multiple MRI sequences and/or manual interaction and specialized expertise. In this study, we develop a robust method for automatic fiducial marker detection in prostate MRI simulation images and quantify the pretreatment alignment accuracy using automatically detected fiducial markers in MRI. In this study, a deep learning-based algorithm was used to convert MRI images into labeled fiducial marker volumes. Seventy-seven prostate cancer patients who received marker implantation prior to MRI and CT simulation imaging were selected for this study. Multiple-Echo T1 -VIBE MRI images were acquired, and images were stratified (at the patient level) based on the presence of intraprostatic calcifications. Ground truth (GT) contours were defined by an expert on MRI using CT images. Training was done using the pix2pix generative adversarial network (GAN) image-to-image translation package and model testing was done using fivefold cross validation. For performance comparison, an experienced medical dosimetrist and a medical physicist each manually contoured fiducial markers in MRI images. The percent of correct detections and F1 classification scores are reported for markers detected using the automatic detection algorithm and human observers. The patient positioning errors were quantified by calculating the target registration errors (TREs) from fiducial marker driven rigid registration between MRI and CBCT images. Target registration errors were quantified for fiducial marker contours defined on MRI by the automatic detection algorithm and the two expert human observers. Ninety-six percent of implanted fiducial markers were correctly identified using the automatic detection algorithm. Two expert raters correctly identified 97% and 96% of fiducial markers, respectively. The F1 classification score was 0.68, 0.75, and 0.72 for the automatic detection algorithm and two human raters, respectively. The main source of false discoveries was intraprostatic calcifications. The mean TRE differences between alignments from automatic detection algorithm and human detected markers and GT were <1mm. We have developed a deep learning-based approach to automatically detect fiducial markers in MRI-only simulation images in a clinically representative patient cohort. The automatic detection algorithm-predicted markers can allow for patient setup with similar accuracy to independent human observers.

A Novel Method of Synovitis Stratification in Ultrasound Using Machine Learning Algorithms: Results From Clinical Validation of the MEDUSA Project

Ultrasound is widely used in the diagnosis and follow-up of chronic arthritis. We present an evaluation of a novel automatic ultrasound diagnostic tool based on image recognition technology. Methods used in developing the algorithm are described elsewhere. For the purpose of evaluation, we collected 140 ultrasound images of metacarpophalangeal and proximal interphalangeal joints from patients with chronic arthritis. They were classified, according to hypertrophy size, into four stages (0–3) by three independent human observers and the algorithm. An agreement ratio was calculated between all observers and the standard derived from results of human staging using κ statistics. Results was significant in all pairs, with the highest p value of 3.9 × 10–6. κ coefficients were lower in algorithm/human pairs than between human assessors. The algorithm is effective in staging synovitis hypertrophy. It is, however, not mature enough to use in a daily practice because of limited accuracy and lack of color Doppler recognition. These limitations will be addressed in the future.

Location Sensitive Deep Convolutional Neural Networks for Segmentation of White Matter Hyperintensities

The anatomical location of imaging features is of crucial importance for accurate diagnosis in many medical tasks. Convolutional neural networks (CNN) have had huge successes in computer vision, but they lack the natural ability to incorporate the anatomical location in their decision making process, hindering success in some medical image analysis tasks. In this paper, to integrate the anatomical location information into the network, we propose several deep CNN architectures that consider multi-scale patches or take explicit location features while training. We apply and compare the proposed architectures for segmentation of white matter hyperintensities in brain MR images on a large dataset. As a result, we observe that the CNNs that incorporate location information substantially outperform a conventional segmentation method with handcrafted features as well as CNNs that do not integrate location information. On a test set of 50 scans, the best configuration of our networks obtained a Dice score of 0.792, compared to 0.805 for an independent human observer. Performance levels of the machine and the independent human observer were not statistically significantly different (p-value = 0.06).

O30. The effect of collimator choice on quantitative assessment of Iodine-123 on the grounds of contrast accuracy and heart–mediastinum ratio: A simulated phantom study

Introduction Literature reports much debate about collimator choice for imaging with I-123, depending on whether image quantification or image quality interpretation is required. The aim of this study was to investigate the relative effect of septal penetration due to collimator choice on contrast quantification accuracy and image quality using a segmented and thoracic phantom, simulated using Simind Monte Carlo software. Materials and methods The first part of this study evaluated the amount of septal penetration present when using either a low-energy-high-resolution or medium-energy collimator with Iodine-123 inside a segmented phantom for source-to-collimator distances of 2 cm and 10 cm. Contrast accuracy was determined by correlating the collected counts in a region of interest with the expected number predetermined from the ratio of activities simulated. The second part focused on the clinical implication, where the heart-to-mediastinum ratio is evaluated using I-123 in a thoracic phantom set-up using abovementioned collimators. All images were evaluated for image quality by blinded independent human observers provided with a score sheet. Results In part one, the contrast accuracy proved marginally better by 7.6 ± 2.3% and 16.2 ± 6.2% at 2 and 10 cm respectively when using the medium-energy collimator. In part two, the heart-mediastinum ratio was more accurate with a percentage difference of 4.8% for the medium-energy collimator in comparison to 21.4% for the low-energy-high-resolution collimator. In both part one and two the medium-energy collimator delivered better contrast, whilst the low-energy high-resolution collimator provided better image resolution. Conclusion When conducting a quantitative analysis based Iodine-123 study using contrast ratios in planar images, contrast accuracy should be prioritized above the overall image quality and the medium-energy collimator should be employed. For an Iodine-123 study in which a visual diagnosis needs to be made from the planar images, image resolution should be optimized by using the low-energy high-resolution collimator.

Automatic differentiation of color fundus images containing drusen or exudates using a contextual spatial pyramid approach.

We developed an automatic system to identify and differentiate color fundus images containing no lesions, drusen or exudates. Drusen and exudates are lesions with a bright appearance, associated with age-related macular degeneration and diabetic retinopathy, respectively. The system consists of three lesion detectors operating at pixel-level, combining their outputs using spatial pooling and classification with a random forest classifier. System performance was compared with ratings of two independent human observers using human-expert annotations as reference. Kappa agreements of 0.89, 0.97 and 0.92 and accuracies of 0.93, 0.98 and 0.95 were obtained for the system and observers, respectively.

Clavicle segmentation in chest radiographs

Automated delineation of anatomical structures in chest radiographs is difficult due to superimposition of multiple structures. In this work an automated technique to segment the clavicles in posterior-anterior chest radiographs is presented in which three methods are combined. Pixel classification is applied in two stages and separately for the interior, the border and the head of the clavicle. This is used as input for active shape model segmentation. Finally dynamic programming is employed with an optimized cost function that combines appearance information of the interior of the clavicle, the border, the head and shape information derived from the active shape model. The method is compared with a number of previously described methods and with independent human observers on a large database. This database contains both normal and abnormal images and will be made publicly available. The mean contour distance of the proposed method on 249 test images is 1.1±1.6mm and the intersection over union is 0.86±0.10.

Multi-Atlas-Based Segmentation With Local Decision Fusion—Application to Cardiac and Aortic Segmentation in CT Scans

A novel atlas-based segmentation approach based on the combination of multiple registrations is presented. Multiple atlases are registered to a target image. To obtain a segmentation of the target, labels of the atlas images are propagated to it. The propagated labels are combined by spatially varying decision fusion weights. These weights are derived from local assessment of the registration success. Furthermore, an atlas selection procedure is proposed that is equivalent to sequential forward selection from statistical pattern recognition theory. The proposed method is compared to three existing atlas-based segmentation approaches, namely 1) single atlas-based segmentation, 2) average-shape atlas-based segmentation, and 3) multi-atlas-based segmentation with averaging as decision fusion. These methods were tested on the segmentation of the heart and the aorta in computed tomography scans of the thorax. The results show that the proposed method outperforms other methods and yields results very close to those of an independent human observer. Moreover, the additional atlas selection step led to a faster segmentation at a comparable performance.

Diagnostic image analysis of malignant melanoma in in vivo confocal laser‐scanning microscopy: a preliminary study

In this study we assessed the applicability of image analysis and a machine learning algorithm on diagnostic discrimination of benign and malignant melanocytic skin tumours in in vivo confocal laser-scanning microscopy (CLSM). A total of 857 CLSM tumour images including 408 benign nevi and 449 melanoma images was evaluated. Image analysis was based on features of the wavelet transform. For classification purposes we used a classification tree software (CART). Moreover, automated image analysis results were compared with the prediction success of an independent human observer. CART analysis of the whole set of CLSM tumour images correctly classified 97.55% and 96.32% of melanoma and nevi images. In contrast, sensitivity and specificity of 85.52% and 80.15% could be reached by the human observer. When the image set was randomly divided into a learning (67% of the images) and a test set (33% of the images), overall 97.31% and 81.03% of the tumour images in the learning and test set could be classified correctly by the CART procedure. Provided automated decisions can be used as a second opinion. This can be valuable in assisting diagnostic decisions in this new and exciting imaging technique.

Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public database

The task of segmenting the lung fields, the heart, and the clavicles in standard posterior–anterior chest radiographs is considered. Three supervised segmentation methods are compared: active shape models, active appearance models and a multi-resolution pixel classification method that employs a multi-scale filter bank of Gaussian derivatives and a k-nearest-neighbors classifier. The methods have been tested on a publicly available database of 247 chest radiographs, in which all objects have been manually segmented by two human observers. A parameter optimization for active shape models is presented, and it is shown that this optimization improves performance significantly. It is demonstrated that the standard active appearance model scheme performs poorly, but large improvements can be obtained by including areas outside the objects into the model. For lung field segmentation, all methods perform well, with pixel classification giving the best results: a paired t-test showed no significant performance difference between pixel classification and an independent human observer. For heart segmentation, all methods perform comparably, but significantly worse than a human observer. Clavicle segmentation is a hard problem for all methods; best results are obtained with active shape models, but human performance is substantially better. In addition, several hybrid systems are investigated. For heart segmentation, where the separate systems perform comparably, significantly better performance can be obtained by combining the results with majority voting. As an application, the cardio-thoracic ratio is computed automatically from the segmentation results. Bland and Altman plots indicate that all methods perform well when compared to the gold standard, with confidence intervals from pixel classification and active appearance modeling very close to those of a human observer. All results, including the manual segmentations, have been made publicly available to facilitate future comparative studies.

A new method for the quantitative analysis of sleep spindles during continuous overnight EEG recordings

Sleep spindles are one of the most characteristic EEG transients of sleep. Quantitative analysis of sleep spindles may be of value in the study of involuntary movement disorders, depression, and sleep disorders, but this is difficult to perform manually. Phase-locked loop based devices have previously been used to analyse preselected periods of artefact-free non-REM sleep. We have developed a device of this type that provides reliable quantitative analysis of sleep spindles during continuous overnight EEG recordings. The number of sleep spindles detected by the device during overnight recordings correlated well with the mean of two independent human observers and measurements of the amplitude and duration of individual sleep spindles were similar to those obtained by visual analysis. The general applicability of the device was demonstrated by studying 61 patients with a range of neurological disorders.

Signs of intelligence in cross-fostered chimpanzees.

In cross-fostering, the young of one species are reared by adults of another, as in the classical ethological studies of imprinting and song-learning. In our laboratory, infant chimpanzees were reared under human conditions that included two-way communication in American Sign Language (A.S.L.), the gestural language of the deaf in North America. A large body of evidence from five chimpanzees demonstrated stage by stage replication of basic aspects of the acquisition of speech and signs by hearing and deaf children. Here we review evidence that, under double-blind conditions: (i) the chimpanzees communicated information in A.S.L. to human observers; (ii) independent human observers agreed in their identification of the chimpanzee signs, (iii) the chimpanzees could use the signs to refer to natural language categories: DOG for any dog, FLOWER for any flower, SHOE for any shoe.

A vocabulary test for chimpanzees (Pan troglodytes).

Chimpanzees can communicate in American Sign Language (ASL) to independent human observers whose only source of information is the ASL signs of the chimpanzees. A vocabulary test was presented to 4 cross-fostered chimpanzees (4-6 years old). Thirty-five-millimeter color slides were projected on a screen that could be seen by the chimpanzee subject but not by the human observers. There were two observers: O1 was the questioner in the testing room with the subject; O2 was in a different room. Neither observer could see the other, or the responses of the other observer. O1 and O2 agreed in their readings of both correct and incorrect signs, and most of the signs were the correct ASL names of the slides. In order to show that the chimpanzees were naming natural language categories--that the sign DOG could refer to any dog, FLOWER to any flower, SHOE to any shoe--each test trial was a first trial in that test slides were presented only once. Analysis of errors showed that two aspects of the signs, gestural form and conceptual category, governed the distribution of errors.