Poor Imaging Conditions Research Articles

The New Frontiers of Fetal Imaging: MRI Insights into Cardiovascular and Thoracic Structures.

Fetal magnetic resonance imaging (fMRI) represents a second-line imaging modality that provides multiparametric and multiplanar views that are crucial for confirming diagnoses, detecting associated pathologies, and resolving inconclusive ultrasound findings. The introduction of high-field magnets and new imaging sequences has expanded MRI's role in pregnancy management. Recent innovations in ECG-gating techniques have revolutionized the prenatal evaluation of congenital heart disease by synchronizing imaging with the fetal heartbeat, thus addressing traditional challenges in cardiac imaging. Fetal cardiac MRI (fCMR) is particularly valuable for assessing congenital heart diseases, especially when ultrasound is limited by poor imaging conditions. fCMR allows for detailed anatomical and functional evaluation of the heart and great vessels and is also useful for diagnosing additional anomalies and analyzing blood flow patterns, which can aid in understanding abnormal fetal brain growth and placental perfusion. This review emphasizes fMRI's potential in evaluating cardiac and thoracic structures, including various gating techniques like metric optimized gating, self-gating, and Doppler ultrasound gating. The review also covers the use of static and cine images for structural and functional assessments and discusses advanced techniques like 4D-flow MRI and T1 or T2 mapping for comprehensive flow quantification and tissue characterization.

A Vision-Based Pose Estimation of a Non-Cooperative Target Based on a Self-Supervised Transformer Network

In the realm of non-cooperative space security and on-orbit service, a significant challenge is accurately determining the pose of abandoned satellites using imaging sensors. Traditional methods for estimating the position of the target encounter problems with stray light interference in space, leading to inaccurate results. Conversely, deep learning techniques require a substantial amount of training data, which is especially difficult to obtain for on-orbit satellites. To address these issues, this paper introduces an innovative binocular pose estimation model based on a Self-supervised Transformer Network (STN) to achieve precise pose estimation for targets even under poor imaging conditions. The proposed method generated simulated training samples considering various imaging conditions. Then, by combining the concepts of convolutional neural networks (CNN) and SIFT features for each sample, the proposed method minimized the disruptive effects of stray light. Furthermore, the feedforward network in the Transformer employed in the proposed method was replaced with a global average pooling layer. This integration of CNN’s bias capabilities compensates for the limitations of the Transformer in scenarios with limited data. Comparative analysis against existing pose estimation methods highlights the superior robustness of the proposed method against variations caused by noisy sample sets. The effectiveness of the algorithm is demonstrated through simulated data, enhancing the current landscape of binocular pose estimation technology for non-cooperative targets in space.

Object Detection for Underwater Cultural Artifacts Based on Deep Aggregation Network with Deformation Convolution

Cultural artifacts found underwater are located in complex environments with poor imaging conditions. In addition, the artifacts themselves present challenges for automated object detection owing to variations in their shape and texture caused by breakage, stacking, and burial. To solve these problems, this paper proposes an underwater cultural object detection algorithm based on the deformable deep aggregation network model for autonomous underwater vehicle (AUV) exploration. To fully extract the object feature information of underwater objects in complex environments, this paper designs a multi-scale deep aggregation network with deformable convolutional layers. In addition, the approach also incorporates a BAM module for feature optimization, which enhances the potential feature information of the object while weakening the background interference. Finally, the object prediction is achieved through feature fusion at different scales. The proposed algorithm has been extensively validated and analyzed on the collected underwater artifact datasets, and the precision, recall, and mAP of the algorithm have reached 93.1%, 91.4%, and 92.8%, respectively. In addition, our method has been practically deployed on an AUV. In the field testing over a shipwreck site, the artifact detection frame rate reached up to 18 fps, which satisfies the real-time object detection requirement.

Improving Scene Text Image Super-resolution via Dual Prior Modulation Network

Scene text image super-resolution (STISR) aims to simultaneously increase the resolution and legibility of the text images, and the resulting images will significantly affect the performance of downstream tasks. Although numerous progress has been made, existing approaches raise two crucial issues: (1) They neglect the global structure of the text, which bounds the semantic determinism of the scene text. (2) The priors, e.g., text prior or stroke prior, employed in existing works, are extracted from pre-trained text recognizers. That said, such priors suffer from the domain gap including low resolution and blurriness caused by poor imaging conditions, leading to incorrect guidance. Our work addresses these gaps and proposes a plug-and-play module dubbed Dual Prior Modulation Network (DPMN), which leverages dual image-level priors to bring performance gain over existing approaches. Specifically, two types of prior-guided refinement modules, each using the text mask or graphic recognition result of the low-quality SR image from the preceding layer, are designed to improve the structural clarity and semantic accuracy of the text, respectively. The following attention mechanism hence modulates two quality-enhanced images to attain a superior SR result. Extensive experiments validate that our method improves the image quality and boosts the performance of downstream tasks over five typical approaches on the benchmark. Substantial visualizations and ablation studies demonstrate the advantages of the proposed DPMN. Code is available at: https://github.com/jdfxzzy/DPMN.

DPF-S2S: A novel dual-pathway-fusion-based sequence-to-sequence text recognition model

In this paper, a novel dual-pathway-fusion-based sequence-to-sequence learning model (DPF-S2S) is proposed for text recognition in the wild, which mainly focuses on enriching the spatial information and extracting high-dimensional representation features to assist decoding. In particular, a double alignment module is developed to solve the problem of text misalignment, where both position and vision information are well considered. Moreover, a global fusion module is deployed to enrich 2D information in the aligned attention maps, which benefits accurate recognition from complicated scenes with arbitrary text shapes and poor imaging conditions. Benchmark evaluations on seven datasets have demonstrated the superiority of proposed DPF-S2S model in comparison to other state-of-the-art text recognition methods, which presents great competitiveness on identifying texts in both regular and irregular scenes. In addition, extensive ablation studies have been carried out, which validate the effectiveness of applied strategies in proposed DPF-S2S.

Model-Agnostic Temporal Regularizer for Object Localization Using Motion Fields

Video analysis often requires locating and tracking target objects. In some applications, the localization system has access to the full video, which allows fine-grain motion information to be estimated. This paper proposes capturing this information through motion fields and using it to improve the localization results. The learned motion fields act as a model-agnostic temporal regularizer that can be used with any localization system based on keypoints. Unlike optical flow-based strategies, our motion fields are estimated from the model domain, based on the trajectories described by the object keypoints. Therefore, they are not affected by poor imaging conditions. The benefits of the proposed strategy are shown on three applications: 1) segmentation of cardiac magnetic resonance; 2) facial model alignment; and 3) vehicle tracking. In each case, combining popular localization methods with the proposed regularizer leads to improvement in overall accuracies and reduces gross errors.

Updates to and Performance of the cBathy Algorithm for Estimating Nearshore Bathymetry from Remote Sensing Imagery

This manuscript describes and tests a set of improvements to the cBathy algorithm, published in 2013 by Holman et al. [hereafter HPH13], for the estimation of bathymetry based on optical observations of propagating nearshore waves. Three versions are considered, the original HPH13 algorithm (now labeled V1.0), an intermediate version that has seen moderate use but limited testing (V1.2), and a substantially updated version (V2.0). Important improvements from V1.0 include a new deep-water weighting scheme, removal of a spurious variable in the nonlinear fitting, an adaptive scheme for determining the optimum tile size based on the approximate wavelength, and a much-improved search seed algorithm. While V1.2 was tested and results listed, the primary interest is in comparing V1.0, the original code, with the new version V2.0. The three versions were tested against an updated dataset of 39 ground-truth surveys collected from 2015 to 2019 at the Field Research Facility in Duck, NC. In all, 624 cBathy collections were processed spanning a four-day period up to and including each survey date. Both the unfiltered phase 2 and the Kalman-filtered phase 3 bathymetry estimates were tested. For the Kalman-filtered estimates, only the estimate from mid-afternoon on the survey date was used for statistical measures. Of those 39 Kalman products, the bias, rms error, and 95% exceedance for V1.0 were 0.15, 0.47, and 0.96 m, respectively, while for V2.0, they were 0.08, 0.38, and 0.78 m. The mean observed coverage, the percentage of successful estimate locations in the map, were 99.1% for V1.0 and 99.9% for V2.0. Phase 2 (unfiltered) bathymetry estimates were also compared to ground truth for the 624 available data runs. The mean bias, rms error, and 95% exceedance statistics for V1.0 were 0.19, 0.64, and 1.27 m, respectively, and for V2.0 were 0.16, 0.56, and 1.19 m, an improvement in all cases. The coverage also increased from 78.8% for V1.0 to 84.7% for V2.0, about a 27% reduction in the number of failed estimates. The largest errors were associated with both large waves and poor imaging conditions such as fog, rain, or darkness that greatly reduced the percentage of successful coverage. As a practical mitigation of large errors, data runs for which the significant wave height was greater than 1.2 m or the coverage was less than 50% were omitted from the analysis, reducing the number of runs from 624 to 563. For this reduced dataset, the bias, rms error, and 95% exceedance errors for V1.0 were 0.15, 0.58, and 1.16 m and for V2.0 were 0.09, 0.41, and 0.85 m, respectively. Successful coverage for V1.0 was 82.8%, while for V2.0, it was 90.0%, a roughly 42% reduction in the number of failed estimates. Performance for V2.0 individual (non-filtered) estimates is slightly better than the Kalman results in the original HPH13 paper, and it is recommended that version 2.0 becomes the new standard algorithm.

An enhanced CNN-enabled learning method for promoting ship detection in maritime surveillance system

The accurate and real-time detection of moving ships has become an essential component in maritime video surveillance, leading to enhanced traffic safety and security. With the rapid development of artificial intelligence, it becomes feasible to develop intelligent techniques to promote ship detection results in maritime applications. In this work, we propose to develop an enhanced convolutional neural network (CNN) to improve ship detection under different weather conditions. To be specific, the learning and representation capacities of our network are promoted by redesigning the sizes of anchor boxes, predicting the localization uncertainties of bounding boxes, introducing the soft non-maximum suppression, and reconstructing a mixed loss function. In addition, a flexible data augmentation strategy with generating synthetically-degraded images is presented to enlarge the volume and diversity of original dataset to train learning-based ship detection methods. This strategy is capable of making our CNN-based detection results more reliable and robust under adverse weather conditions, e.g., rain, haze, and low illumination. Experimental results under different monitoring conditions demonstrate that our method significantly outperforms other competing methods (e.g., SSD, Faster R-CNN, YOLOv2 and YOLOv3) in terms of detection accuracy, robustness and efficiency. The ship detection results under poor imaging conditions have also been implemented to demonstrate the superior performance of our learning method.

Deep multisensor learning for missing-modality all-weather mapping

Multisensor Earth observation has significantly accelerated the development of multisensor collaborative remote sensing applications such as all-weather mapping using synthetic aperture radar (SAR) images and optical images. However, in the real-world application scenarios, not all data sources may be available, namely, the missing-modality problem, e.g., the poor imaging conditions obstruct the optical sensors, and only SAR images are available for the mapping. This real-world scenario raises the challenge of how to leverage historical multisensor data to improve the representation ability of the available model. The knowledge transfer based and knowledge distillation based approaches, as feasible solutions, can be used to transfer knowledge from other sensor models to the available model. However, these approaches suffer from the problem of forgotten knowledge and the multi-modality co-registration problem, which means that the leveraging of historical multisensor data is inefficient. The essential problem lies in the fact that these approaches are designed following the single-sensor data-driven approach. In this paper, a registration-free multisensor data-driven learning method, namely, deep multisensor learning, in a new perspective of knowledge retention, is proposed to overcome the above problems by learning a meta-sensory representation. To explore the existence of the meta-sensory representation, the meta-sensory representation hypothesis is first proposed, which reveals that the essential difference of the deep models trained on data from different sensors lies in the parameter distribution of the sensor-invariant and sensor-specific operations. Based on this hypothesis, a prototype network is proposed to learn the meta-sensory representation by modeling the knowledge retention mechanism, using the proposed difference alignment operation (DiffAlignOp). DiffAlignOp enables the prototype network to dynamically generate sensor-specific networks to gather supervised signals from registration-free multisensor data. This dynamic network generation is differentiable. Therefore, multisensor gradients can be obtained to learn the meta-sensory representation. To demonstrate the flexibility and practicality of deep multisensor learning, the application of all-weather mapping in a missing-modality scenario was performed. The experiments were conducted on a large public multisensor all-weather mapping dataset, which consists of high-resolution optical and SAR imagery with a spatial resolution of 0.5 m. The experimental results suggest that deep multisensor learning is superior to the other learning approaches in performance and stability, and reveals the importance of meta-sensory representation in multisensor remote sensing applications.

Toward a general model for reflection recovery and single image enhancement

Images often suffer from low visual quality due to poor imaging conditions such as low light or hazy weather. The haze imaging model is widely used in contrast enhancement in daylight condition with haze, while the retinex model is universal for low-light conditions. Although their forms and applications are different, they can be unified into a more general form through the proposed observation. Based on this model, the authors can estimate the reflection of the scene more accurately in more complex imaging conditions. In this study, the authors propose a simple but effective method for estimating the reflection and enhancing the image contrast based on a general imaging model. To preserve the image details and control contrast, the authors introduce dark boundary and bright boundary to handle the high-light and low-light conditions, and a guided structure-preserving optimization algorithm is proposed to estimate them. After obtaining the dark and bright boundaries, the reflection is calculated and the image is enhanced accordingly. Different from previous approaches, which were designed for specific applications, the proposed method can be used for more diverse imaging conditions. Experiments show that the proposed method can be applied to many poor imaging conditions and maintain good performance.

Robust Edge Detection Method for Segmentation of Diabetic Foot Ulcer Images

Segmentation is an open-ended research problem in various computer vision and image processing tasks. This pre-processing operation requires a robust edge detector to generate appealing results. However, the available approaches for edge detection underperform when applied to images corrupted by noise or impacted by poor imaging conditions. The problem becomes significant for images containing diabetic foot ulcers, which originate from people with varied skin color. Comparative performance evaluation of the edge detectors facilitates the process of deciding an appropriate method for image segmentation of diabetic foot ulcers. Our research discovered that the classical edge detectors cannot clearly locate ulcers in images with black-skin feet. In addition, these methods collapse for degraded input images. Therefore, the current research proposes a robust edge detector that can address some limitations of the previous attempts. The proposed method incorporates a hybrid diffusion-steered functional derived from the total variation and the Perona-Malik diffusivities, which have been reported to can effectively capture semantic features in images. The empirical results show that our method generates clearer and stronger edge maps with higher perceptual and objective qualities. More importantly, the proposed method offers lower computational times—an advantage that gives more insights into the possible application of the method in time-sensitive tasks.

Pose detection in complex classroom environment based on improved Faster R‐CNN

Pose detection of small targets in poor imaging conditions like heavy occlusion and low resolution is still an open and challenging task in computer vision. For instance, detection of students' poses in classrooms that are even indistinguishable to human eyes remains a rather difficult task. Motivated by the success of convolutional feature merging and locality preserving, the authors propose a pose detection framework combining merged region of interest (ROI) pooling and locality preserving learning. Unlike usual object detection algorithms which use general top-level convolutional features as inputs, their method uses a merged ROI pooling structure to merge semantic feature and high-resolution feature from the last two levels of convolutional feature maps, so that this merged feature is made more expressive than the single-level feature. In addition, the locality feature-preserving learning is used in the last fully-connected layer. Through locality preserving learning, features belonging to the same class would be forced to be closer in the feature space, which enables the model with stronger classification ability. Experimental results show that the proposed method outperforms the state-of-the-art methods.

Motion prediction via online instantaneous frequency estimation for vision-based beating heart tracking

The beating heart tracking based on stereo endoscope remains challenging due to highly dynamic scenes and poor imaging conditions in minimally invasive surgery. This paper proposes a new prediction method for robust tracking of heart motion. The dual time-varying Fourier series is used for modeling the motion of points of interest (POI) on heart surfaces, which is driven jointly by breathing and heartbeat motion. A dual Kalman filtering scheme is used to estimate the frequencies and Fourier coefficients of the model respectively. A novel orthogonal decomposition algorithm is developed to measure the instantaneous frequencies of breathing and heartbeat motion online from the 3D trajectory of the POI. The difference in direction between breathing and heartbeat motion is exploited by using principal component analysis on the past trajectory, and optimal 1D principal component signals are extracted for measuring the corresponding frequencies. The frequencies calculated from the orthogonal subbands are fused based on an additive noise model for optimal frequency measurement. The proposed method is evaluated and compared with other available prediction methods based on the simulated data and the real-measured signals from the videos recorded by the daVinci® surgical robot. The prediction algorithm is finally incorporated into a well-established visual tracking method to handle long-term occlusions.

Robust edge detector based on anisotropic diffusion-driven process

Edge detection involves a process to discriminate, highlight, and extract useful image features (edges and contours). In many situations, we prefer an edge detector that distinguishes these features more accurately, and which comfortably deals with a variety of data. Our observations, however, discovered that most edge-defining functionals underperform and generate false edges under poor imaging conditions. Therefore, the current research proposes a robust diffusion-driven edge detector for seriously degraded images. The method is iterative, and suppresses noise while simultaneously marking real edges and deemphasizing false edges. The anisotropic nature of the new functional helps to remove noise and to preserve semantic structures. Even more importantly, the functional exhibits a forward–backward behavior that may sharpen and strengthen edges. Comparisons with some other classical approaches demonstrate superiority of the proposed approach.

Unsteady camera zoom stabilization using slope estimation over interest warping vectors

Projections being based on global image representative are considered as simplified and efficient stabilization solutions under poor textural or imaging conditions. The existing projection algorithms are investigated only for translation or rotation stabilization, but the image captured with unsteady camera also suffers from undesired zoom-in/out effect. This paper presents a new zoom estimation algorithm for extended projection applicability. Camera focusing generally being effective around its optical axis, produces zooming around the image center. This image-centered zoom results in an opposite relative motion (stretching/compression) in two halves of the image projections. A novel concept of utilizing this relative longitudinal projection motion is proposed for camera zoom estimation. Derivative dynamic time warping is used for image projection alignment and the effect of mismatched-warping singularities leading to motion estimation error is removed by selecting some key warping vectors labeled as Interest Warping Vectors. Slope values of these interest warping vectors in the two projection-halves are utilized for zoom factor estimation and the approach is also generalized for arbitrary zoom-center under forward translation. Robustness of the proposed algorithm tested with artificial textural artifacts is verified with motion accuracy analysis on various sequentially zoomed test and real-world images within 0.8 to 1.2 scale factor range.

Fluorescence and SEM correlative microscopy for nanomanipulation of subcellular structures

Nanomanipulation under scanning electron microscopy (SEM) enables direct interactions of a tool with a sample. We recently developed a nanomanipulation technique for the extraction and identification of DNA contained within sub-nuclear locations of a single cell nucleus. In nanomanipulation of sub-cellular structures, a key step is to identify targets of interest through correlating fluorescence and SEM images. The DNA extraction task must be conducted with low accelerating voltages resulting in low imaging resolutions. This is imposed by the necessity of preserving the biochemical integrity of the sample. Such poor imaging conditions make the identification of nanometer-sized fiducial marks difficult. This paper presents an affine scale-invariant feature transform (ASIFT) based method for correlating SEM images and fluorescence microscopy images. The performance of the image correlation approach under different noise levels and imaging magnifications was quantitatively evaluated. The optimal mean absolute error (MAE) of correlation results is 68±34 nm under standard conditions. Compared with manual correlation by skilled operators, the automated correlation approach demonstrates a speed that is higher by an order of magnitude. With the SEM-fluorescence image correlation approach, targeted DNA was successfully extracted via nanomanipulation under SEM conditions. A fast method for correlating scanning electron microscopy (SEM) and fluorescence images when handling subcellular objects has been acheived. Yu Sun of the University of Toronto, Canada, and co-workers in China and Canada developed the image correlation technique to improve the implementation of a method for extracting DNA from cells. Such correlation is essential, because SEM images lack sufficient contrast to enable structures of interest to be identified and because of the need to use low-energy (and hence low-resolution) SEM imaging to avoid damaging DNA. Their correlation technique employs image processing algorithms to automatically correlate the two microscopy techniques and is an order of magnitude faster than manual image correlation by skilled operators. The researchers have demonstrated the method's usefulness by using it to extract DNA from cells under SEM imaging.

3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model

Visual tracking techniques based on stereo endoscope are developed to measure tissue motion in robot-assisted minimally invasive surgery. However, accurate 3D tracking of tissue surfaces remains challenging due to complicated deformation, poor imaging conditions, specular reflections and other dynamic effects during surgery. This study employs a robust and efficient 3D tracking scheme with two independent recursive processes, namely kernel-based inter-frame motion estimation and model-based intra-frame 3D matching. In the first process, target region is represented in joint spatial-color space for robust estimation. By defining a probabilistic similarity measure, a mean-shift-based iterative algorithm is derived for location of the target region in a new image. In the second process, the thin-plate spline model is used to fit the 3D shape of tissue surfaces around the target region. An iterative algorithm based on an efficient second-order minimization technique is derived to compute optimal model parameters. The two processes can be computed in parallel. Their outputs are combined to recover 3D information about the target region. The performance of the proposed method is validated using phantom heart videos and in vivo videos acquired by the daVinci® surgical robotic platform and a synthesized data set with known ground truth.

Increased Robustness of Single-Molecule Counting with Microfluidics, Digital Isothermal Amplification, and a Mobile Phone versus Real-Time Kinetic Measurements

Quantitative bioanalytical measurements are commonly performed in a kinetic format and are known to not be robust to perturbation that affects the kinetics itself or the measurement of kinetics. We hypothesized that the same measurements performed in a "digital" (single-molecule) format would show increased robustness to such perturbations. Here, we investigated the robustness of an amplification reaction (reverse-transcription loop-mediated amplification, RT-LAMP) in the context of fluctuations in temperature and time when this reaction is used for quantitative measurements of HIV-1 RNA molecules under limited-resource settings (LRS). The digital format that counts molecules using dRT-LAMP chemistry detected a 2-fold change in concentration of HIV-1 RNA despite a 6 °C temperature variation (p-value = 6.7 × 10(-7)), whereas the traditional kinetic (real-time) format did not (p-value = 0.25). Digital analysis was also robust to a 20 min change in reaction time, to poor imaging conditions obtained with a consumer cell-phone camera, and to automated cloud-based processing of these images (R(2) = 0.9997 vs true counts over a 100-fold dynamic range). Fluorescent output of multiplexed PCR amplification could also be imaged with the cell phone camera using flash as the excitation source. Many nonlinear amplification schemes based on organic, inorganic, and biochemical reactions have been developed, but their robustness is not well understood. This work implies that these chemistries may be significantly more robust in the digital, rather than kinetic, format. It also calls for theoretical studies to predict robustness of these chemistries and, more generally, to design robust reaction architectures. The SlipChip that we used here and other digital microfluidic technologies already exist to enable testing of these predictions. Such work may lead to identification or creation of robust amplification chemistries that enable rapid and precise quantitative molecular measurements under LRS. Furthermore, it may provide more general principles describing robustness of chemical and biological networks in digital formats.

Multiple Hypothesis Tracking for Cluttered Biological Image Sequences

In this paper, we present a method for simultaneously tracking thousands of targets in biological image sequences, which is of major importance in modern biology. The complexity and inherent randomness of the problem lead us to propose a unified probabilistic framework for tracking biological particles in microscope images. The framework includes realistic models of particle motion and existence and of fluorescence image features. For the track extraction process per se, the very cluttered conditions motivate the adoption of a multiframe approach that enforces tracking decision robustness to poor imaging conditions and to random target movements. We tackle the large-scale nature of the problem by adapting the multiple hypothesis tracking algorithm to the proposed framework, resulting in a method with a favorable tradeoff between the model complexity and the computational cost of the tracking procedure. When compared to the state-of-the-art tracking techniques for bioimaging, the proposed algorithm is shown to be the only method providing high-quality results despite the critically poor imaging conditions and the dense target presence. We thus demonstrate the benefits of advanced Bayesian tracking techniques for the accurate computational modeling of dynamical biological processes, which is promising for further developments in this domain.

Enhancement for road sign images and its performance evaluation

Recently, road sign recognition in natural scenes for intelligent transportation system (ITS) applications are paid much attention to, such as license plate recognition, traffic sign recognition and road text recognition. To improve recognition performance under some poor imaging conditions, e.g. complex lighting variations and pollution, an enhancement method for road sign images is proposed in this paper. It is composed of two steps: contrast maximizing while graying and mathematical morphological tophat–bothat transform. Performance evaluation results have demonstrated that the propose method can do well in purposiveness, real-time performance, and empirical parameter independence simultaneously, comparing to some other enhancement methods.