Pixel Alignment Research Articles

Seam estimation based on dense matching for parallax-tolerant image stitching

Image stitching with large parallax poses a significant challenge in the field of computer vision. Existing seam-based approaches attempt to address parallax artifacts by stitching images along seams. However, issues such as object mismatches, disappearances, and duplications still arise occasionally, primarily due to inaccurate alignment of dense pixels or inappropriate seam estimation methods. In this paper, we propose a robust seam-based parallax-tolerant image stitching method that leverages dense flow estimation from state-of-the-art approaches. Firstly, we develop a seam estimation method that does not require pre-estimation of image warping model. Instead, it directly estimates the seam by measuring the local smoothness of the optical flow field and incorporating a penalty term for duplications. Subsequently, we design an iterative algorithm that utilizes the location of estimated seam to solve a spatial smooth warping model and eliminate outlier corresponding pairs. By employing this approach, we effectively address the intertwined challenges of estimating the warping model and seam. Experiment on real-world images shows that our proposed method achieves superior local alignment accuracy near the stitching seam and outperforms other state-of-the-art techniques on visual stitching result.

An Implicit Geometric Optimization Study of Intelligent Teaching Methods in the Reform of Public Physical Education Courses in Colleges and Universities

Abstract Virtual reality is an important realization of intelligent teaching methods, which is of great significance for the reform of public sports courses in colleges and universities. Aiming at the problem of the lack of three-dimensional information representation of the sports human body in virtual reality technology, this paper proposes a representation optimization method based on implicit geometry, which improves the traditional representation by pixel alignment features, voxel alignment features, and the introduction of z-difference. Experiments show that the new representation method has smaller representation errors and higher computational speeds. The table tennis forehand and backhand table tennis scores of the experimental group are higher than those of the control group by 7.8% and 12.5%, and the total score of the match ability reaches 71.05. The interest score is higher than that of the control group by 3.23 points. The positive effects of the new representation and intelligent teaching methods are evident.

A boundary enhancement and pixel alignment based smoke segmentation network

Image segmentation methods usually fuse shallow and deep features to locate object boundaries, but it is difficult to improve the accuracy of smoke segmentation by conventional fusion methods. It is a very difficult vision task to perform semantic segmentation of smoke images, because the translucency and irregular shapes of smoke lead to extremely complicated mixtures with background that are adverse to segmentation. To improve the segmentation accuracy of smoke scenes, we propose a Boundary Enhancement and Pixel Alignment based smoke segmentation Network for fire alarms. For the shallow features of the network, an attention mechanism is adopted to capture spatially details of smoke for improving boundary precision. For the deep layers, the Pyramid Pooling Module is used to extract local features and abstract semantic ones simultaneously. Finally, to efficiently merge shallow and deep features, a Pixel Alignment Module is adopted to model the relationship between pixel locations. The experimental results show that the mean Intersection over Union of the proposed method on the three synthetic smoke test datasets is 78.61%, 77.63% and 77.30%, respectively, and it outperforms most of the existing methods. In addition, our method obtains satisfying results on inconspicuous smoke and smoke-like images.

Delving Deep into Pixel Alignment Feature for Accurate Multi-View Human Mesh Recovery

Regression-based methods have shown high efficiency and effectiveness for multi-view human mesh recovery. The key components of a typical regressor lie in the feature extraction of input views and the fusion of multi-view features. In this paper, we present Pixel-aligned Feedback Fusion (PaFF) for accurate yet efficient human mesh recovery from multi-view images. PaFF is an iterative regression framework that performs feature extraction and fusion alternately. At each iteration, PaFF extracts pixel-aligned feedback features from each input view according to the reprojection of the current estimation and fuses them together with respect to each vertex of the downsampled mesh. In this way, our regressor can not only perceive the misalignment status of each view from the feedback features but also correct the mesh parameters more effectively based on the feature fusion on mesh vertices. Additionally, our regressor disentangles the global orientation and translation of the body mesh from the estimation of mesh parameters such that the camera parameters of input views can be better utilized in the regression process. The efficacy of our method is validated in the Human3.6M dataset via comprehensive ablation experiments, where PaFF achieves 33.02 MPJPE and brings significant improvements over the previous best solutions by more than 29%. The project page with code and video results can be found at https://kairobo.github.io/PaFF/.

Reverse-Net: Few-Shot Learning with Reverse Teaching for Deformable Medical Image Registration

Multimodal medical image registration has an important role in monitoring tumor growth, radiotherapy, and disease diagnosis. Deep-learning-based methods have made great progress in the past few years. However, its success depends on large training datasets, and the performance of the model decreases due to overfitting and poor generalization when only limited data are available. In this paper, a multimodal medical image registration framework based on few-shot learning is proposed, named reverse-net, which can improve the accuracy and generalization ability of the network by using a few segmentation labels. Firstly, we used the border enhancement network to enhance the ROI (region of interest) boundaries of T1 images to provide high-quality data for the subsequent pixel alignment stage. Secondly, through a coarse registration network, the T1 image and T2 image were roughly aligned. Then, the pixel alignment network generated more smooth deformation fields. Finally, the reverse teaching network used the warped T1 segmentation labels and warped images generated by the deformation field to teach the border enhancement network more structural knowledge. The performance and generalizability of our model have been evaluated on publicly available brain datasets including the MRBrainS13DataNii-Pro, SRI24, CIT168, and OASIS datasets. Compared with VoxelMorph, the reverse-net obtained performance improvements of 4.36% in DSC on the publicly available MRBrainS13DataNii-Pro dataset. On the unseen dataset OASIS, the reverse-net obtained performance improvements of 4.2% in DSC compared with VoxelMorph, which shows that the model can obtain better generalizability. The promising performance on dataset CIT168 indicates that the model is practicable.

Face Shape-Guided Deep Feature Alignment for Face Recognition Robust to Face Misalignment

For the past decades, face recognition (FR) has been actively studied in computer vision and pattern recognition society. Recently, due to the advances in deep learning, the FR technology shows high performance for most of the benchmark datasets. However, when the FR algorithm is applied to a real-world scenario, the performance has been known to be still unsatisfactory. This is mainly attributed to the mismatch between training and testing sets. Among such mismatches, face misalignment between training and testing faces is one of the factors that hinder successful FR. To address this limitation, we propose a face shape-guided deep feature alignment framework for FR robust to the face misalignment. Based on a face shape prior (e.g., face keypoints), we train the proposed deep network by introducing alignment processes, i.e., pixel and feature alignments, between well-aligned and misaligned face images. Through the pixel alignment process that decodes the aggregated feature extracted from a face image and face shape prior, we add the auxiliary task to reconstruct the well-aligned face image. Since the aggregated features are linked to the face feature extraction network as a guide via the feature alignment process, we train the robust face feature to the face misalignment. Even if the face shape estimation is required in the training stage, the additional face alignment process, which is usually incorporated in the conventional FR pipeline, is not necessarily needed in the testing phase. Through the comparative experiments, we validate the effectiveness of the proposed method for the face misalignment with the FR datasets.

Controlling StyleGANs using rough scribbles via one‐shot learning

AbstractThis paper tackles the challenging problem of one‐shot semantic image synthesis from rough sparse annotations, which we call “semantic scribbles.” Namely, from only a single training pair annotated with semantic scribbles, we generate realistic and diverse images with layout control over, for example, facial part layouts and body poses. We present a training strategy that performs pseudo labeling for semantic scribbles using the StyleGAN prior. Our key idea is to construct a simple mapping between StyleGAN features and each semantic class from a single example of semantic scribbles. With such mappings, we can generate an unlimited number of pseudo semantic scribbles from random noise to train an encoder for controlling a pretrained StyleGAN generator. Even with our rough pseudo semantic scribbles obtained via one‐shot supervision, our method can synthesize high‐quality images thanks to our GAN inversion framework. We further offer optimization‐based postprocessing to refine the pixel alignment of synthesized images. Qualitative and quantitative results on various datasets demonstrate improvement over previous approaches in one‐shot settings.

An automatic method and comprehensive analysis of inter-waveband pixel alignment across the Sloan Digital Sky Survey

Comparing a galaxy’s structure across multiple wavebands may require high spatial resolution. For example, determining the color gradient across a spiral arm may allow us to ascertain the co-rotation radius, which can in turn inform theoretical considerations on arm formation and evolution. However, pixels in the Sloan Digital Sky Survey are not precisely aligned across wavebands, making unbiased pixel-resolution comparisons difficult. Since foreground stars are effectively point sources, their location can be determined to sub-pixel precision by fitting their light profiles to an appropriate point spread function. Here we describe a completely automated method that uses foreground stars in the Sloan Digital Sky Survey to align images of galaxies across wavebands to about 0.02 pixels. We have applied the method to the approximately 900,000 galaxies collected from the Galaxy Zoo 1 project. The technique is not specific to the Sloan Digital Sky Survey and could be easily adaptable to other surveys, provided they only require translation for alignment. We have made all source code and aligned images publicly available.

Crossmodality Person Reidentification Based on Global and Local Alignment

RGB-infrared (RGB-IR) person reidentification is a challenge problem in computer vision due to the large crossmodality difference between RGB and IR images. Most traditional methods only carry out feature alignment, which ignores the uniqueness of modality differences and is difficult to eliminate the huge differences between RGB and IR. In this paper, a novel AGF network is proposed for RGB-IR re-ID task, which is based on the idea of global and local alignment. The AGF network distinguishes pedestrians in different modalities globally by combining pixel alignment and feature alignment and highlights more structure information of person locally by weighting channels with SE-ResNet-50, which has achieved ideal results. It consists of three modules, including alignGAN module ( A ), crossmodality paired-images generation module ( G ), and feature alignment module ( F ). First, at pixel level, the RGB images are converted into IR images through the pixel alignment strategy to directly reduce the crossmodality difference between RGB and IR images. Second, at feature level, crossmodality paired images are generated by exchanging the modality-specific features of RGB and IR images to perform global set-level and fine-grained instance-level alignment. Finally, the SE-ResNet-50 network is used to replace the commonly used ResNet-50 network. By automatically learning the importance of different channel features, it strengthens the ability of the network to extract more fine-grained structural information of person crossmodalities. Extensive experimental results conducted on SYSU-MM01 dataset demonstrate that the proposed method favorably outperforms state-of-the-art methods. In addition, we evaluate the performance of the proposed method on a stronger baseline, and the evaluation results show that a RGB-IR re-ID method will show better performance on a stronger baseline.

SPRTracker: Learning Spatial-Temporal Pixel Aggregations for Multiple Object Tracking

Recently, multiple object tracking based on a single frame has achieved excellent performance. However, in crowded scenes, occlusion and motion blur will increase the difficulty of foreground object detection. In this letter, we propose a Spatial-temporal Pixel Resampling Tracker (SPRTracker) that introduces a novel cross-frame input framework to improve the anti-occlusion and anti-interference ability during tracking. We first propose a sampling mechanism in which Inter-frame Pixel Alignment (IPA) is designed to maintain spatial and temporal consistency in the propagation of pixel-level information between frames, aiming to improve the anti-interference ability during tracking target motion. Secondly, To improve the compensation effect of the gain information in the historical frame to the occluded target in the current frame, Similarity Reparameter Fusion (SRF) strategy is designed to fuse the features of the two frames. We evaluated the proposed method on two common benchmarks (MOT17 and MOT20), and the experimental results effectively demonstrated the superiority of our method.

3D Pedestrian Detection in Farmland by Monocular RGB Image and Far-Infrared Sensing

The automated driving of agricultural machinery is of great significance for the agricultural production efficiency, yet is still challenging due to the significantly varied environmental conditions through day and night. To address operation safety for pedestrians in farmland, this paper proposes a 3D person sensing approach based on monocular RGB and Far-Infrared (FIR) images. Since public available datasets for agricultural 3D pedestrian detection are scarce, a new dataset is proposed, named as “FieldSafePedestrian”, which includes field images in both day and night. The implemented data augmentations of night images and semi-automatic labeling approach are also elaborated to facilitate the 3D annotation of pedestrians. To fuse heterogeneous images of sensors with non-parallel optical axis, the Dual-Input Depth-Guided Dynamic-Depthwise-Dilated Fusion network (D5F) is proposed, which assists the pixel alignment between FIR and RGB images with estimated depth information and deploys a dynamic filtering to guide the heterogeneous information fusion. Experiments on field images in both daytime and nighttime demonstrate that compared with the state-of-the-arts, the dynamic aligned image fusion achieves an accuracy gain of 3.9% and 4.5% in terms of center distance and BEV-IOU, respectively, without affecting the run-time efficiency.

An Automatic Optimized Method for a Digital Optical Phase Conjugation System in Focusing through Scattering Media

In this paper, a reliable automatic optimized method for a digital optical phase conjugation (DOPC) system based on a multipopulation genetic algorithm (MPGA) is proposed for improving the compensation quality of DOPC. The practical implementation and compensation quality of DOPC in focusing through scattering media are greatly limited by imperfect pixel alignment, optical aberration, and mechanical error in the DOPC system. For comprehensively solving the above problems, the concept of global optimization is introduced by Zernike polynomials (Zernike modes) to characterize overall imperfections, and MPGA is used to search for the most optimal Zernike coefficient and compensate for the overall imperfections of the DOPC system. The significant optimization ability of the proposed method is verified in DOPC-related experiments for focusing through scattering media. The peak-to-background ratio (PBR) of the OPC focus increases 174 times that of the initial OPC focus. Furthermore, we evaluated the optimization results of the proposed method with a fitness function of intensity fitness and correlation coefficient fitness in MPGA. The results show that the optimized capability is excellent and more efficiently used than the correlation coefficient fitness function in the Zernike modes.

Plaintext attack on joint transform correlation encryption system by convolutional neural network.

The image encryption system based on joint transform correlation has attracted much attention because its ciphertext does not contain complex value and can avoid strict pixel alignment of ciphertext when decryption occurs. This paper proves that the joint transform correlation architecture is vulnerable to the attack of the deep learning method-convolutional neural network. By giving the convolutional neural network a large amount of ciphertext and its corresponding plaintext, it can simulate the key of the encryption system. Unlike the traditional method which uses the phase recovery algorithm to retrieve or estimate optical encryption key, the key model trained in this paper can directly convert the ciphertext to the corresponding plaintext. Compared with the existing neural network systems, this paper uses the sigmoid activation function and adds dropout layers to make the calculation of the neural network more rapid and accurate, and the equivalent key trained by the neural network has certain robustness. Computer simulations prove the feasibility and effectiveness of this method.

Abstract 6584: The ‘AstroPath' platform for spatially resolved, single cell analysis of the tumor microenvironment (TME) using multispectral immunofluorescence (mIF)

Abstract Background: Multidimensional, spatially resolved analyses of immune and tumor cells within the TME of patients treated with checkpoint inhibitors will provide clinically translatable mechanistic insights and potentiate biomarker discovery. To achieve this goal, information from pathology specimens needs to be captured at a single cell level with high fidelity and in meaningfully sized cohorts. To date, efforts have been limited by inadequate tissue sampling and previously unrecognized errors in staining, imaging and data analysis. Here we describe the ‘AstroPath' platform, where strategies from the field of astronomy were adapted to study pathology specimens and generate large high quality mIF data. Methods: Potential error was identified and addressed at each stage of 6-plex (PD-1, PD-L1, FoxP3, CD163, CD8, tumor marker) mIF assay development. Whole slides from formalin-fixed paraffin embedded tissue specimens were stained with the optimized assay and imaged using a multispectral microscope (Vectra 3.0) with 20% overlap of high power fields (HPFs). The overlaps were used to quantify and correct optical lens distortion, HPF alignment, and illumination variation. Errors from cell segmentation algorithms, batch-to-batch staining variation, and HPF sampling were also addressed. The resultant mIF data were organized and analyzed using a large, relational database. Results: The optimized mIF assay captured equivalent signal compared to gold standard chromogenic immunohistochemistry and 2x more signal for PD-1, PD-L1 and FoxP3 compared to the manufacturer's recommended protocol. Errors and corrections for imaging included: pixel alignment error reduced from ~10 to <0.5 pixels at edges of HPFs; illumination variation reduced from 10% to 3% per HPF; over-counting of larger cells, e.g. tumor cells, reduced by ~25% using custom cell segmentation and ‘multi-pass' phenotyping algorithms; and batch-to-batch variation reduced by ~50% by normalizing to tissue controls. Correction of these errors that would otherwise be compounded at each stage, allowed for more accurate and reliable cell type and marker intensity comparisons across samples. Lastly, the entire slide rather than select HPFs were imaged, resulting in ~100x more HPFs analyzed per slide. The whole slide imaging approach also corrected for other potential source of errors, i.e., sampling error due to tumor heterogeneity and operator-dependent field selection. Conclusion: Here we present an end-to-end pathology workflow with rigorous quality control for creating quantitative, spatially resolved mIF datasets using lessons derived from the field of astronomy. Such approaches will vastly improve standardization and scalability of mIF technologies, enabling cross-site comparisons and eventual clinical translation as biomarker discovery platforms or standard diagnostic tests. Citation Format: Sneha Berry, Nicolas Giraldo, Benjamin Green, Elizabeth Engle, Haiying Xu, Aleksandra Ogurtsova, Daphne Wang, Julie E. Stein, Peter Nguyen, Suzanne Topalian, Angelo DeMarzo, Drew M. Pardoll, Robert A. Anders, Tricia R. Cottrell, Alexander S. Szalay, Janis M. Taube. The ‘AstroPath' platform for spatially resolved, single cell analysis of the tumor microenvironment (TME) using multispectral immunofluorescence (mIF) [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 6584.

Localization-aware channel pruning for object detection

Channel pruning is one of the important methods for deep model compression. Most of existing pruning methods mainly focus on classification. Few of them conduct systematic research on object detection. However, object detection is different from classification, which requires not only semantic information but also localization information. In this paper, based on discrimination-aware channel pruning (DCP) which is state-of-the-art pruning method for classification, we propose a localization-aware auxiliary network to find out the channels with key information for classification and regression so that we can conduct channel pruning directly for object detection, which saves lots of time and computing resources. In order to capture the localization information, we first design the auxiliary network with a contextual RoIAlign layer which can obtain precise localization information of the default boxes by pixel alignment and enlarges the receptive fields of the default boxes when pruning shallow layers. Then, we construct a loss function for object detection task which tends to keep the channels that contain the key information for classification and regression. Extensive experiments demonstrate the effectiveness of our method. On MS COCO, we prune 70% parameters of the SSD based on ResNet-50 with modest accuracy drop, which outperforms the-state-of-art method.

A Novel Image Fusion Scheme using Wavelet Transform for Concealed Weapon Detection

The aim of this paper is to detect concealed weapons, especially in high security places like in airports, train stations and places with large crowds, where concealed weapons are not allowed. We aim to specify suspicious person who may have a concealed weapon. In this paper, an Image Fusion technique using pixel alignment and discrete wavelet transform is proposed. It is mainly utilized for Concealed Weapon Detection. Image fusion can be defined as extracting information from two or images into a single image to enhance the detection. Image fusion allows detecting concealed weapons underneath a person’s clothing with imaging sensors such as Infrared imaging or Passive Millimeter Wave sensors. A data fusion scheme for simpler sensors based on correlation coefficients is proposed and utilized. We proposed an image fusion scheme that utilizes fusion dependency rules using wavelet (WT) and inverse wavelet transform (IWT). The fusion rule is to select the coefficient with the highest correlation rate. The higher the correlation the stronger of the co-existed feature. Experimental results shows the superiority of the proposed algorithm both in quality and real time requirement. The proposed algorithm has a real time response time that is less than other comparable algorithms by 40%. At the same time it retains higher quality as shown in the experimental results. It outperforms other algorithms by superior PSNR of more than 10% of the comparable algorithms in average.

Tectonic and Anthropogenic Deformation at the Cerro Prieto Geothermal Step-Over Revealed by Sentinel-1A InSAR

The Cerro Prieto geothermal field (CPGF) lies at the step-over between the imperial and the Cerro Prieto faults in northern Baja California, Mexico. While tectonically this is the most active section of the southern San Andreas Fault system, the spatial and temporal deformation in the area is poorly resolved by the sparse global positioning system (GPS) network coverage. Moreover, interferograms from satellite observations spanning more than a few months are decorrelated due to the extensive agricultural activity in this region. Here we investigate the use of frequent, short temporal baseline interferograms offered by the new Sentinel-1A satellite to recover two components of deformation time series across these faults. Following previous studies, we developed a purely geometric approach for image alignment that achieves better than 1/200 pixel alignment needed for accurate phase recovery. We construct interferometric synthetic aperture radar time series using a coherence-based small baseline subset method with atmospheric corrections by means of common-point stacking. We did not apply enhanced spectral diversity because the burst discontinuities are generally small (<1.4 mm) and can be effectively captured during the atmospheric corrections. With these algorithms, the subsidence at CPGF is clearly resolved. The maximum subsidence rate of 160 mm/yr, due to extraction of geothermal fluids and heat, dominates the ~40 mm/yr deformation across the proximal ends of the imperial, the Cerro Prieto, and the indiviso faults.

GridPix detectors – introduction and applications

GridPix detectors are a new kind of detectors combining a high density pixelized readout ASIC with a Micromegas gas amplification stage. Because of the alignment of mesh holes and pixels, a high efficiency for detecting and separating single primary electrons is reached. This feature leads to excellent spatial and energy resolutions as demonstrated by several different setups.

Performance of correlation approaches for the evaluation of spatial distortion reductions

The analysis of optical remote sensing images often requires a perfect pixel alignment between single bands. Even smallest deviations may degrade the accuracy of subsequent parameter retrieval or lead to the detection of non-existing structures caused by artificial gradients. Hence, a careful pre-processing is essential for minimizing spatial non-uniformities such as erroneous co-registration. The results need to be validated and assigned with a quality flag that is unfortunately still not a common practice. In this letter, four broadly used global correlation approaches, a two-dimensional Gaussian peak fit, a poly phase technique, an iterative phase approach and its proposed enhancement, were tested for their capacity to serve either as an evaluation tool for preceding spatial distortion reductions, e.g. by co-registration, or as a global minimizer for generic reduction approaches. For this eight broadly used test images, three Landsat 7 and three Landsat 8 samples were sub pixel shifted artificially and degraded by different noise levels resulting in more than 200,000 noise and shift scenarios. Additionally, one state-of-the-art approach was enhanced by 50% on average for all scenarios and by 280% on average for all non-degraded images. This study indicates that three out of four approaches can serve as evaluation tools for spatial distortion reductions or as a global minimizer even for highly degraded images, whereas proposed enhancement offers highest accuracy and the poly phase approach offers best overall performance with regard to considered criteria.

3D dynamic holographic display by modulating complex amplitude experimentally

Complex amplitude modulation method is presented theoretically and performed experimentally for three-dimensional (3D) dynamic holographic display with reduced speckle using a single phase-only spatial light modulator. The determination of essential factors is discussed based on the basic principle and theory. The numerical simulations and optical experiments are performed, where the static and animated objects without refinement on the surfaces and without random initial phases are reconstructed successfully. The results indicate that this method can reduce the speckle in reconstructed images effectively; furthermore, it will not cause the internal structure in the reconstructed pixels. Since the complex amplitude modulation is based on the principle of phase-only hologram, it does not need the stringent alignment of pixels. This method can be used for high resolution imaging or measurement in various optical areas.