Pairwise Image Comparisons Research Articles

Intelligent robotic sonographer: Mutual information-based disentangled reward learning from few demonstrations

Ultrasound (US) imaging is widely used for biometric measurement and diagnosis of internal organs due to the advantages of being real-time and radiation-free. However, due to inter-operator variations, resulting images highly depend on the experience of sonographers. This work proposes an intelligent robotic sonographer to autonomously “explore” target anatomies and navigate a US probe to standard planes by learning from the expert. The underlying high-level physiological knowledge from experts is inferred by a neural reward function, using a ranked pairwise image comparison approach in a self-supervised fashion. This process can be referred to as understanding the “language of sonography.” Considering the generalization capability to overcome inter-patient variations, mutual information is estimated by a network to explicitly disentangle the task-related and domain features in latent space. The robotic localization is carried out in coarse-to-fine mode based on the predicted reward associated with B-mode images. To validate the effectiveness of the proposed reward inference network, representative experiments were performed on vascular phantoms (“line” target), two types of ex vivo animal organ phantoms (chicken heart and lamb kidney representing “point” target), and in vivo human carotids. To further validate the performance of the autonomous acquisition framework, physical robotic acquisitions were performed on three phantoms (vascular, chicken heart, and lamb kidney). The results demonstrated that the proposed advanced framework can robustly work on a variety of seen and unseen phantoms as well as in vivo human carotid data. Code: https://github.com/yuan-12138/MI-GPSR. Video: https://youtu.be/u4ThAA9onE0.

Measuring and modelling perceptions of the built environment for epidemiological research using crowd-sourcing and image-based deep learning models.

Perceptions of the built environment, such as nature quality, beauty, relaxation, and safety, may be key factors linking the built environment to human health. However, few studies have examined these types of perceptions due to the difficulty in quantifying them objectively in large populations. To measure and predict perceptions of the built environment from street-view images using crowd-sourced methods and deep learning models for application in epidemiologic studies. We used the Amazon Mechanical-Turk crowdsourcing platform where participants compared two street-view images and quantified perceptions of nature quality, beauty, relaxation, and safety. We optimized street-view image sampling methods to improve the quality and resulting perception data specific to participants enrolled in the Washington State Twin Registry (WSTR) health study. We used a transfer learning approach to train deep learning models by leveraging existing image perception data from the PlacePulse 2.0 dataset, which includes 1.1 million image comparisons, and refining based on new WSTR perception data. Resulting models were applied to WSTR addresses to estimate exposures and evaluate associations with traditional built environment measures. We collected over 36,000 image comparisons and calculated perception measures for each image. Our final deep learning models explained 77.6% of nature quality, 68.1% of beauty, 72.0% of relaxation, and 64.7% of safety in pairwise image comparisons. Applying transfer learning with the new perception labels specific to the WSTR yielded an average improvement of 3.8% for model performance. Perception measures were weakly to moderately correlated with traditional built environment exposures for WSTR participant addresses; for example, nature quality and NDVI (r = 0.55), neighborhood area deprivation (r = -0.16), and walkability (r = -0.20), respectively. We were able to measure and model perceptions of the built environment optimized for a specific health study. Future applications will examine associations between these exposure measures and mental health in the WSTR. Built environments influence health through complex pathways. Perceptions of nature quality, beauty, relaxation and safety may be particularly import for understanding these linkages, but few studies to-date have examined these perceptions objectively for large populations. For quantitative research, an exposure measure must be reproducible, accurate, and precise--here we work to develop such measures for perceptions of the urban environment. We created crowd-sourced and image-based deep learning methods that were able to measure and model these perceptions. Future applications will apply these models to examine associations with mental health in the Washington State Twin Registry.

Social media user identity linkage by graphic content comparison

The article proposes a new approach to comparing accounts of the social media “VKontakte” and “Instagram” to determine those accounts which belong to the same user. The approach is based on the comparison of graphic content; the novelty of the approach consists in combining several methods for matching graphic content, also for the first time a method is proposed for matching accounts of the mentioned social media. The proposed method combines three methods of matching graphic content: by extracting the faces of the account users from the photos in the account and matching them, by matching all faces in both accounts, and by pairwise comparison of images to determine the same images in both accounts using the perceptual pHash method. The described method was tested on a dataset of more than 8,000 pairs of accounts. According to the results of the experiment, the value of the F1-score metric reached 0.87. The practical significance lies in automating the comparison of user accounts in various social networks by implementing of the developed algorithm in the prototype of the software package. A further direction for research lies in expanding the set of data and attributes of profiles considered for comparison. The results can be introduced into a software package for the analysis of the security of a user of information systems against social engineering attacks. It seems promising to combine the obtained findings with account matching methods based on the structural similarity of social graphs.

Closed Reduction for the Treatment of Grade IV Supination-External Rotation Fracture of the Ankle Joint: A Retrospective Analysis.

ObjectiveTo investigate the curative effect of closed reduction and external fixation in the treatment of grade IV supination‐external rotation fractures of the ankle joint.MethodsFifty‐six patients treated with closed reduction and external fixation from February 2016 to March 2020 were included in this retrospective study, all with sprains. After receiving nerve block anesthesia, the patient underwent closed reduction under C‐arm fluoroscopy, and the ankle joint was fixed in a dorsiflexion‐inversion position with casting and splints after the end of the fracture met the reduction standard by fluoroscopy. One week and four weeks after the reduction treatment, oblique axial and coronal MR scans of the ankle joint were performed to determine the degree of injury and healing of the inferior tibiofibular syndesmosis; anteroposterior and lateral X‐rays of the ankle joint (including the ankle acupoints) were regularly reviewed to observe the fracture alignment and healing. Combined with the images and physical examination, the patients were instructed to undergo ankle weight‐bearing rehabilitation training when they met the clinical healing standard, and at the last follow‐up, the Mazur ankle evaluation and grading system were used for evaluation. After the reduction, the images were evaluated according to the Leeds standard. The image healing of fracture was evaluated by callus growth criteria.ResultsThe follow‐up period of patients ranged from 11 to 58 months, with an average of 26.8 months. The clinical healing time was (8.51 ± 2.12) weeks. The excellent and good rating after reduction was 82.1%, and the excellent and good rating during clinical fracture healing was 73.2%, according to the Leeds imaging evaluation. According to the Mazur ankle evaluation and grading system, the excellent and good rating was 75.0%. Pairwise comparison of callus images at 4, 6 and 12 weeks showed statistically significant differences (P < 0.05), suggesting callus growth at different time periods. A total of 56 patients had anterior inferior tibial fibular ligament (AITFL) injuries (grade II‐III), among which 11 patients had AITFL injuries combined with grade II injuries of the interosseous ligament (IOL) and 4 patients had AITFL injuries combined with grade III injuries of the IOL.ConclusionsMost of the patients with grade IV supination‐external rotation fracture of the ankle joint had good prognosis after closed reduction and plaster combined with splint fixation. For patients with IOL injury who had poor prognosis, open reduction and internal fixation therapy is appropriate.

Discovering Image Manipulation History by Pairwise Relation and Forensics Tools

Given a potentially manipulated probe image, provenance analysis aims to find all images derived from the probe (offsprings) and all images from which the probe is derived (ancestors) in a large dataset (provenance filtering), and reconstruct the manipulation history with the retrieved images (provenance graph building). In this paper, we address two major challenges in provenance analysis, retrieving the source image of the small regions that are spliced into the probe image, and, detecting source images within the search results. For the former challenge, we propose to detect spliced regions by pairwise image comparison and only use local features extracted from the spliced region to perform an additional search. This removes the influence of the background and greatly improves the recall. For the latter, we propose to learn a pairwise ancestor-offspring detector and use it jointly with a holistic image manipulation detector to identify the source image. The proposed provenance analysis system has performed remarkably in evaluations using comprehensive provenance datasets. It's the winning solution for NIST Media Forensics Challenge (MFC) in 2018, 2019 and 2020. In MFC 2019, our provenance results achieved a 12% improvement in filtering and a 20% gain in oracle provenance graphs building over the alternative methods. In the real-world Reddit dataset, the edge overlap between our reconstructed provenance graphs and the ground-truth graphs is 5 times better than the state-of-the-art system.

Yum-Me: A Personalized Nutrient-Based Meal Recommender System.

Nutrient-based meal recommendations have the potential to help individuals prevent or manage conditions such as diabetes and obesity. However, learning people’s food preferences and making recommendations that simultaneously appeal to their palate and satisfy nutritional expectations are challenging. Existing approaches either only learn high-level preferences or require a prolonged learning period. We propose Yum-me, a personalized nutrient-based meal recommender system designed to meet individuals’ nutritional expectations, dietary restrictions, and fine-grained food preferences. Yum-me enables a simple and accurate food preference profiling procedure via a visual quiz-based user interface and projects the learned profile into the domain of nutritionally appropriate food options to find ones that will appeal to the user. We present the design and implementation of Yum-me and further describe and evaluate two innovative contributions. The first contriution is an open source state-of-the-art food image analysis model, named FoodDist. We demonstrate FoodDist’s superior performance through careful benchmarking and discuss its applicability across a wide array of dietary applications. The second contribution is a novel online learning framework that learns food preference from itemwise and pairwise image comparisons. We evaluate the framework in a field study of 227 anonymous users and demonstrate that it outperforms other baselines by a significant margin. We further conducted an end-to-end validation of the feasibility and effectiveness of Yum-me through a 60-person user study, in which Yum-me improves the recommendation acceptance rate by 42.63%.

Short-Term Change Detection in Wetlands Using Sentinel-1 Time Series

Automated monitoring systems that can capture wetlands’ high spatial and temporal variability are essential for their management. SAR-based change detection approaches offer a great opportunity to enhance our understanding of complex and dynamic ecosystems. We test a recently-developed time series change detection approach (S1-omnibus) using Sentinel-1 imagery of two wetlands with different ecological characteristics; a seasonal isolated wetland in southern Spain and a coastal wetland in the south of France. We test the S1-omnibus method against a commonly-used pairwise comparison of consecutive images to demonstrate its advantages. Additionally, we compare it with a pairwise change detection method using a subset of consecutive Landsat images for the same period of time. The results show how S1-omnibus is capable of capturing in space and time changes produced by water surface dynamics, as well as by agricultural practices, whether they are sudden changes, as well as gradual. S1-omnibus is capable of detecting a wider array of short-term changes than when using consecutive pairs of Sentinel-1 images. When compared to the Landsat-based change detection method, both show an overall good agreement, although certain landscape changes are detected only by either the Landsat-based or the S1-omnibus method. The S1-omnibus method shows a great potential for an automated monitoring of short time changes and accurate delineation of areas of high variability and of slow and gradual changes.

Effects of thresholding on correlation-based image similarity metrics.

The computation of image similarity is important for a wide range of analyses in neuroimaging, from decoding to meta-analysis. In many cases the images being compared have empty voxels, but the effects of such empty voxels on image similarity metrics are poorly understood. We present a detailed investigation of the influence of different degrees of image thresholding on the outcome of pairwise image comparison. Given a pair of brain maps for which one of the maps is thresholded, we show that an analysis using the intersection of non-zero voxels across images at a threshold of Z = ±1.0 maximizes accuracy for retrieval of a list of maps of the same contrast, and thresholding up to Z = ±2.0 can increase accuracy as compared to comparison using unthresholded maps. Finally, maps can be thresholded up to to Z = ±3.0 (corresponding to 25% of voxels non-empty within a standard brain mask) and still maintain a lower bound of 90% accuracy. Our results suggest that a small degree of thresholding may improve the accuracy of image similarity computations, and that robust meta-analytic image similarity comparisons can be obtained using thresholded images.

Multivariate Hypergeometric Similarity Measure

We propose a similarity measure based on the multivariate hypergeometric distribution for the pairwise comparison of images and data vectors. The formulation and performance of the proposed measure are compared with other similarity measures using synthetic data. A method of piecewise approximation is also implemented to facilitate application of the proposed measure to large samples. Example applications of the proposed similarity measure are presented using mass spectrometry imaging data and gene expression microarray data. Results from synthetic and biological data indicate that the proposed measure is capable of providing meaningful discrimination between samples, and that it can be a useful tool for identifying potentially related samples in large-scale biological data sets.

Subtraction MR Images in a Multiple Sclerosis Multicenter Clinical Trial Setting

To explore the applicability of subtraction magnetic resonance (MR) images to (a) detect active multiple sclerosis (MS) lesions, (b) directly quantify lesion load change, and (c) detect treatment effects (distinguish treatment arms) in a placebo-controlled multicenter clinical trial by comparing the subtraction scheme with a conventional pair-wise comparison of nonregistered MR images. Forty-six pairs of MR studies in 40 patients (31 women; mean age, 31.9 years) from a multicenter clinical trial were used. The clinical trial was approved by local ethics review boards, and all subjects gave written informed consent. Active MS lesions were scored by two independent raters, and lesion load measurements were conducted by using semiautomated software. Lesion counts were evaluated by using the Wilcoxon signed rank test, interrater agreement was evaluated by using the intraclass correlation coefficient (ICC), and treatment (interferon beta-1b) effect was evaluated by using the Mann-Whitney U test. When subtraction images were used, there was a 1.7-fold increase in the detection of positive active lesions, as compared with native image pairs, and significantly greater interobserver agreement (ICC = 0.98 vs 0.91, P < .001). Subtraction images also allowed direct quantification of positive disease activity, a measure that provided sufficient power to distinguish treatment arms (P = .012) compared with the standard measurement of total lesion load change on native images (P = .455). MR image subtraction enabled detection of higher numbers of active MS lesions with greater interobserver agreement and exhibited increased power to distinguish treatment arms, as compared with a conventional pair-wise comparison of nonregistered MR images.

The variability of regional airflow obstruction within the lungs of patients with asthma: Assessment with hyperpolarized helium-3 magnetic resonance imaging

It is unknown whether focal changes of airflow obstruction within the lungs of patients with asthma vary or are fixed in location with time or repeated bronchoconstriction. With hyperpolarized helium-3 magnetic resonance (H(3)HeMR) imaging, the airspaces are depicted and focal areas of airflow obstruction are shown as "ventilation defects." To investigate the regional changes of airflow obstruction with time and repeated bronchoconstriction. H(3)HeMR and spirometry were performed before (pre) and immediately after (post) methacholine challenge in 10 young patients with asthma on 2 days that were 7-476 days (mean, 185.3 +/- 37.2 days) apart. Pair-wise image comparisons were performed to determine the change in location of ventilation defects within the lung and their change in size. When comparing premethacholine versus premethacholine and postmethacholine versus post-methacholine images of the 2 days, 41% +/- 10% and 69% +/- 5% (P = .017) of defects, respectively, were in the same location, and of those, 69% +/- 12% and 43% +/- 5% (P = .022), respectively, did not change size. Comparing premethacholine versus postmethacholine images, 58% +/- 9% of defects were in the same location on day 1 and 73% +/- 7% (P = .088) on day 2. On both days, the percent increase in defect number from premethacholine to postmethacholine was much greater than the percent decrease in spirometric values (P < .001). Many of the ventilation defects persisted or recurred in the same location with time or repeated bronchoconstriction, suggesting that the regional changes of airflow obstruction are relatively fixed within the lung. The findings give new insight into the regional airflow variability within the lungs of patients with asthma.

Twenty-one years of crown area change in the Jasper Ridge Biological Preserve based on georeferenced multitemporal aerial photographs

A time series of large-scale aerial photographs of a 25 ha section of the Jasper Ridge Biological Preserve (JRBP) in California was analysed for the quantification of long-term changes in the orthogonally projected crown areas of a sample of mature canopy trees growing in closed woodland and open grassland habitats. Crown area changes were computed on the basis of pairwise image comparisons for the years 1974, 1980, 1988, 1989, 1990, 1991, 1993 and 1995. Single frame image correction methods were assessed by comparing a high-resolution physical-system based method of photogrammetric orthorectification with a low-resolution method of georeferencing that involved the use of ground control points (GCPs) for the derivation of best-fit statistical transformations. The same crown digitization procedure was used for both methods, and a statistically significant difference was not found between the high-resolution and low-resolution measured crown areas. Crown area changes were not correlated with differences in sun position at the time of aerial photo acquisition nor with differences in the off-nadir distances of each crown in the images being compared. The upper limit of digitizing error was 3.5% based on the mean coefficient of variation plus one standard deviation computed from repeat digitization measures of each crown. Crown area changes exceeding 3.5%in the pairwise image comparisons were attributed to actual crown growth or crown reduction. Expressed as a percentage of the original crown area, the mean annual rate of change for each pairwise sample period was 4.9% yr-1. In absolute terms, the mean annual rate of crown area change over the 21-year period 1974-1995 was 1.4m2 yr-1. The long-term mean annual crown growth rate of the clustered woodland trees (1.3m2yr-1) was not significantly different from the isolated grassland trees (1.5m2 yr-1). The mean annual rates of change for the periods 1974-1980, 1980-1989 and 1991-1993 were similar to the long-term average. Unusually low mean growth rates characterized the drought years of 1989-1990 and 1990-1991, while the relatively wet period of 1993-1995 exhibited the most dramatic increase in crown area at a mean rate of 3.0m2yr-1.

Activation pattern reproducibility: Measuring the effects of group size and data analysis models

The reproducibility of patterns from brain activation experiments has been examined only for suprathreshold spatially localized foci. Scatter plots comparing signal levels across all pairs of Talairach voxels for pairs of functional activation images provide an alternative approach for assessing reproducibility. Image-wide, signal-level reproducibility may be quantitatively summarized using pattern similarity measures such as the Pearson product-moment correlation, rho. Empirical population distributions of rho for many pair-wise image comparisons, generated using statistical resampling techniques, may be used to examine the impact of a wide range of experimental variables. We demonstrate the use of such empirical rho-histograms to measure changes in reproducibility for [(15)O]-water PET scans of a simple motor task as a function of group size and data analysis model.