Ground-truth Ones Research Articles

Extreme time extrapolation capabilities and thermodynamic consistency of physics-inspired neural networks for the 3D microstructure evolution of materials via Cahn–Hilliard flow

Abstract A Convolutional Recurrent Neural Network (CRNN) is trained to reproduce the evolution of the spinodal decomposition process in three dimensions as described by the Cahn–Hilliard equation. A specialized, physics-inspired architecture is proven to provide close accordance between the predicted evolutions and the ground truth ones obtained via conventional integration schemes. The method can accurately reproduce the evolution of microstructures not represented in the training set at a fraction of the computational costs. Extremely long-time extrapolation capabilities are achieved, up to reaching the theoretically expected equilibrium state of the system, consisting of a layered, phase-separated morphology, despite the training set containing only relatively-short, initial phases of the evolution. Quantitative accordance with the decay rate of the free energy is also demonstrated up to the late coarsening stages, proving that this class of machine learning approaches can become a new and powerful tool for the long timescale and high throughput simulation of materials, while retaining thermodynamic consistency and high-accuracy.

Coarse-to-fine mechanisms mitigate diffusion limitations on image restoration

Recent years have witnessed the remarkable performance of diffusion models in various vision tasks. However, for image restoration that aims to recover clear images with sharper details from given degraded observations, diffusion-based methods may fail to recover promising results due to inaccurate noise estimation. Moreover, simple constraining noises cannot effectively learn complex degradation information, which subsequently hinders the model capacity. To solve the above problems, we propose a coarse-to-fine diffusion Transformer (C2F-DFT) to mitigate diffusion limitations mentioned before on image restoration. Specifically, the proposed C2F-DFT contains diffusion self-attention (DFSA) and diffusion feed-forward network (DFN) within a new coarse-to-fine training mechanism. The DFSA and DFN with embedded diffusion steps respectively capture the long-range diffusion dependencies and learn hierarchy diffusion representation to guide the restoration process in different time steps. In the coarse training stage, our C2F-DFT estimates noises and then generates the final clean image by a sampling algorithm. To further improve the restoration quality, we propose a simple yet effective fine training pipeline. It first exploits the coarse-trained diffusion model with fixed steps to generate restoration results, which then would be constrained with corresponding ground-truth ones to optimize the models to remedy the unsatisfactory results affected by inaccurate noise estimation. Extensive experiments show that C2F-DFT significantly outperforms diffusion-based restoration method IR-SDE and achieves competitive performance compared with Transformer-based state-of-the-art methods on 3 tasks, including image deraining, image deblurring, and real image denoising. The source codes and visual results are available at https://github.com/wlydlut/C2F-DFT.

Track vertical irregularity estimation for railway bridges using a novel and lightweight deep-learning architecture

A comprehensive methodology is presented for the estimation of track vertical irregularities of railway bridges from vehicle responses, based on a novel and lightweight multi-layer-perceptron (MLP) deep learning architecture. Firstly, a vehicle–track–bridge interaction (VTBI) model is established for the generation of the datasets of deep learning networks. Secondly, the lightweight deep learning architecture is meticulously designed to identify the track's vertical irregularities. Then, the effectiveness of the proposed technique is validated through examples of a single-span simple bridge and a three-span bridge. Further, the sensitivity of the present method against various factors is investigated, including different combinations of vehicle responses, measurement noise, vehicle speed and different classes of track irregularity. It is confirmed that the identified track irregularities of the railway bridges, regardless of irregularity class and noise level, are in excellent agreement with the ground-truth ones. The increase in vehicle speed to some extent reduces the estimation accuracy of the irregularity. The proposed method has higher identification accuracy and efficiency for longer irregularity sequences of multi-span railway bridges.

Mining actionable concepts in concept lattice using Interestingness Propagation

Mining important conceptual patterns is an essential task for understanding the context and content of complex data in many scientific and engineering applications. While exact relevance indices in Formal Concept Analysis provide accurate importance evaluation that can be used for extracting interesting concepts, they often have expensive algorithmic complexity (e.g., at least quadratic in the lattice size, which is exponential with respect to the context size). This frequently results in a computational bottleneck, rendering these indices inappropriate for large, dense contexts or small contexts with large concepts. In this paper, we introduce Interestingness Propagation (IP), an efficient message passing strategy for identifying actionable concepts based on interesting and uninteresting information available from a very small portion of the concepts in lattices. From a conceptual perspective, IP leverages the lattice local and global conceptual structures to maintain local and global consistency in the interesting or uninteresting labeling information among similar neighborhood concepts. Experiments on synthetic and real-world datasets show that IP can accurately extract interesting concepts that are very competitive with ground truth ones computed using state-of-the-art exact interestingness indices, while being at least three times faster than certain indices like CR and one order of magnitude faster than others like stability.

Transmission Removal from a Single Glass Scene and Its Application in Photographer Identification

In daily life, when taking photos of scenes containing glass, the images of the dominant transmission layer and the weak reflection layer are often blended, which are difficult to be uncoupled. Meanwhile, because the reflection layer contains sufficient important information about the surrounding scene and the photographer, the problem of recovering the weak reflection layer from the mixture image is of importance in surveillance investigations. However, most of the current studies mainly focus on extracting the transmission layer while often ignoring the merit of the reflection layer. To fill that gap, in this paper, we propose a network framework that aims to accomplish two tasks: (1) for general scenes, we attempt to recover reflection layer images that are as close as possible to the ground truth ones, and (2) for scenes containing portraits, we recover the basic contour information of the reflection layer while improving the defects of dim portraits in the reflection layer. Through analyzing the performance exhibited by different levels of feature maps, we present the first transmission removal network based on an image-to-image translation architecture incorporating residual structures. The quality of generated reflection layer images is improved via tailored content and style constraints. We also use the patch generative adversarial network to increase the discriminator’s ability to perceive the reflection components in the generated images. Meanwhile, the related information such as edge and color distribution of transmission layer in the mixture image is used to assist the overall reflection layer recovery. In the large-scale experiments, our proposed model outperforms reflection removal-based SOTAs by more than 5.356 dB in PSNR, 0.116 in SSIM, and 0.057 in LPIPS.

A CNN-based no reference image quality metric exploiting content saliency

Assessing the quality of images is a challenging task. To achieve this goal, images must be evaluated by a pool of subjects following a well-defined protocol or an objective quality metric must be defined. In this work, an objective quality metric based on deep neural network is proposed. The metric takes into account the human vision system by computing the saliency map and natural scene statistics features of the image under test. The neural network is composed by two modules: the convolutional layers and the regression units. The first one is trained by using preprocessed distorted images. The feature weights of the first module are smoothed by exploiting the estimated saliency map. The latter module is fit with the ground truth quality scores of the input image and the scaled feature weights obtained from first module by using visual sensitivity factor of image obtained using natural scene statistics features. The performances of the proposed metric have been evaluated by using four datasets: LIVEIQA, TID2013, CSIQ, and KADID10K. The achieved results show the effectiveness of the proposed system in closely matching the predicted quality scores with the ground truth ones.

Unsupervised learning of haptic material properties.

When touching the surface of an object, its spatial structure translates into a vibration on the skin. The perceptual system evolved to translate this pattern into a representation that allows to distinguish between different materials. Here, we show that perceptual haptic representation of materials emerges from efficient encoding of vibratory patterns elicited by the interaction with materials. We trained a deep neural network with unsupervised learning (Autoencoder) to reconstruct vibratory patterns elicited by human haptic exploration of different materials. The learned compressed representation (i.e., latent space) allows for classification of material categories (i.e., plastic, stone, wood, fabric, leather/wool, paper, and metal). More importantly, classification performance is higher with perceptual category labels as compared to ground truth ones, and distances between categories in the latent space resemble perceptual distances, suggesting a similar coding. Crucially, the classification performance and the similarity between the perceptual and the latent space decrease with decreasing compression level. We could further show that the temporal tuning of the emergent latent dimensions is similar to properties of human tactile receptors.

Automatically extracting surfaces of reinforced concrete bridges from terrestrial laser scanning point clouds

Three-dimensional (3D) geometric bridge models play an important role in bridge inspection, assessment, and management. Laser scanning nowadays offers a cost-efficient method to capture dense, accurate 3D topographic data of surfaces of the bridge. However, given the typical complexity of bridges, current workflows using commercial software to construct a bridge model still require intensive labour work. This paper presents a new approach to automatically extract the point cloud of surfaces of structural components of box and slab-beam bridges. The proposed method consists of 3 Parts: (1) point-to-surface, (2) superstructure and (3) substructure extraction. The method uses both spatial point clouds and contextual knowledge to extract point cloud subsets corresponding to surfaces of individual bridge components in a consecutive order from superstructure to substructure. For each bridge component, two levels of extraction are (1) coarse extraction to separate candidate points of the component from the full data set and (2) fine filtering to obtain final 3D points of individual surfaces using cell- or voxel-based region growing (CRG or VRG), followed by a connected surface component (CSC) method. An experimental test on one box-girder and one slab-beam bridges shows that the proposed method successfully extracts all surfaces of bridge components with the lowest F1-score of 0.93 based on a point-based evaluation. Moreover, a shape similarity evaluation also shows that discrepancies between extracted surfaces and ground truth ones are no larger than 0.82 for the area overlap ratio and 0.59 degrees for the angular deviation. The proposed method contributes to the automatic generation of 3D geometric bridge models and to give point clouds of individual surface for damage identification.

Masked Contrastive Representation Learning for Reinforcement Learning.

In pixel-based reinforcement learning (RL), the states are raw video frames, which are mapped into hidden representation before feeding to a policy network. To improve sample efficiency of state representation learning, recently, the most prominent work is based on contrastive unsupervised representation. Witnessing that consecutive video frames in a game are highly correlated, to further improve data efficiency, we propose a new algorithm, i.e., masked contrastive representation learning for RL (M-CURL), which takes the correlation among consecutive inputs into consideration. In our architecture, besides a CNN encoder for hidden presentation of input state and a policy network for action selection, we introduce an auxiliary Transformer encoder module to leverage the correlations among video frames. During training, we randomly mask the features of several frames, and use the CNN encoder and Transformer to reconstruct them based on context frames. The CNN encoder and Transformer are jointly trained via contrastive learning where the reconstructed features should be similar to the ground-truth ones while dissimilar to others. During policy evaluation, the CNN encoder and the policy network are used to take actions, and the Transformer module is discarded. Our method achieves consistent improvements over CURL on 14 out of 16 environments from DMControl suite and 23 out of 26 environments from Atari 2600 Games. The code is available at https://github.com/teslacool/m-curl.

MA-GANet: A Multi-Attention Generative Adversarial Network for Defocus Blur Detection.

Background clutters pose challenges to defocus blur detection. Existing approaches often produce artifact predictions in background areas with clutter and relatively low confident predictions in boundary areas. In this work, we tackle the above issues from two perspectives. Firstly, inspired by the recent success of self-attention mechanism, we introduce channel-wise and spatial-wise attention modules to attentively aggregate features at different channels and spatial locations to obtain more discriminative features. Secondly, we propose a generative adversarial training strategy to suppress spurious and low reliable predictions. This is achieved by utilizing a discriminator to identify predicted defocus map from ground-truth ones. As such, the defocus network (generator) needs to produce 'realistic' defocus map to minimize discriminator loss. We further demonstrate that the generative adversarial training allows exploiting additional unlabeled data to improve performance, a.k.a. semi-supervised learning, and we provide the first benchmark on semi-supervised defocus detection. Finally, we demonstrate that the existing evaluation metrics for defocus detection generally fail to quantify the robustness with respect to thresholding. For a fair and practical evaluation, we introduce an effective yet efficient AUFβ metric. Extensive experiments on three public datasets verify the superiority of the proposed methods compared against state-of-the-art approaches.

Deep learning-based pixel-level rock fragment recognition during tunnel excavation using instance segmentation model

Timely recognition of rock fragments and their morphological sizes can allow adjustment of excavation parameters during tunnel boring machine (TBM) tunnelling. Traditional manual inspection strongly relies on the experience and subjective determination of the human operators, and sieving tests cannot be conducted in real time and are energy-consuming. Rock fragments in real-world images are frequently observed against a dark background, distributed with a high size diversity, and blocked by each other. A novel deep learning-based method is proposed in this paper to achieve real-time on-site rock fragment recognition. The proposed instance segmentation model comprises two subnetworks: object detection and semantic segmentation. The object detection subnetwork is designed based on a modified single-shot detector architecture, and multilevel feature fusion, prior anchors, and self-attention modules are utilised to localise rock fragment regions. The semantic segmentation subnetwork is designed based on U-net. Down-sampling stages use the structures of the object detection subnetwork to share the extracted features of the rock fragments, and up-sampling stages employ skip connection and self-attention modules to accomplish binary segmentation in each detected bounding box. A total of 50 original images with a resolution of 4096 × 3072 were collected: 35 for training and 15 for testing. The results showed that 88.8% of the rock fragments can be recognised and that the average recall and average intersection-over-union values reach 0.87 and 0.76, respectively. Small rock fragments inevitably missed in the labelling process and extremely large ones can also be recognised. The predicted size distributions of the rock fragments fit well with the ground truth ones. Ablation experiments were conducted to further demonstrate the effectiveness of the proposed method. This study presents both visual recognition and statistical results of the size distribution of rock fragments during TBM tunnelling, which can assist in the prediction of rock properties and the adjustment of excavation parameters.

Joint optic disc and optic cup segmentation based on boundary prior and adversarial learning.

The most direct means of glaucoma screening is to use cup-to-disc ratio via colour fundus photography, the first step of which is the precise segmentation of the optic cup (OC) and optic disc (OD). In recent years, convolution neural networks (CNN) have shown outstanding performance in medical segmentation tasks. However, most CNN-based methods ignore the effect of boundary ambiguity on performance, which leads to low generalization. This paper is dedicated to solving this issue. In this paper, we propose a novel segmentation architecture, called BGA-Net, which introduces an auxiliary boundary branch and adversarial learning to jointly segment OD and OC in a multi-label manner. To generate more accurate results, the generative adversarial network is exploited to encourage boundary and mask predictions to be similar to the ground truth ones. Experimental results show that our BGA-Net system achieves state-of-the-art OC and OD segmentation performance on three publicly available datasets, i.e., the Dice scores for the optic disc/cup on the Drishti-GS, RIM-ONE-r3 and REFUGE datasets are 0.975/0.898, 0.967/0.872 and 0.951/0.866, respectively. In this work, we not only achieve superior OD and OC segmentation results, but also confirm that the values calculated through the geometric relationship between the former two are highly related to glaucoma.

Synthesis of Micro-Doppler Signatures of Human Activities From Different Aspect Angles Using Generative Adversarial Networks

In this paper, we propose to produce synthesized micro-Doppler signatures from different aspect angles through conditional generative adversarial networks (cGANs). Micro-Doppler signatures of non-rigid human body motions vary considerably as a function of the radar's aspect angle. Because the direction of the human motion can be arbitrary, a large volume of training data across diverse aspects is needed for practical target activity classification through machine learning. As measurements can require significant monetary and labor costs, the synthesis of micro-Doppler signatures can be an alternate solution. Therefore, we investigate the feasibility of data augmentation through synthesizing micro-Doppler signatures of human activities from diverse radar aspect angles with input data from a single aspect angle. For the training data, the micro-Doppler radar signatures of 12 human activities are generated from different angles ranging from 0 to 315 degrees, at 45-degree increments, through simulations. For each angle, cGANs are trained to synthesize the micro-Doppler signatures for that specific angle given micro-Doppler signatures from another angle. The output of each model is evaluated by calculating mean-square errors and structural similarity indexes between the synthesized micro-Doppler signatures and the ground-truth ones obtained from simulations. We test three different scenarios, and report the respective results.

Sketch-Guided Scenery Image Outpainting.

The outpainting results produced by existing approaches are often too random to meet users' requirements. In this work, we take the image outpainting one step forward by allowing users to harvest personal custom outpainting results using sketches as the guidance. To this end, we propose an encoder-decoder based network to conduct sketch-guided outpainting, where two alignment modules are adopted to impose the generated content to be realistic and consistent with the provided sketches. First, we apply a holistic alignment module to make the synthesized part be similar to the real one from the global view. Second, we reversely produce the sketches from the synthesized part and encourage them be consistent with the ground-truth ones using a sketch alignment module. In this way, the learned generator will be imposed to pay more attention to fine details and be sensitive to the guiding sketches. To our knowledge, this work is the first attempt to explore the challenging yet meaningful conditional scenery image outpainting. We conduct extensive experiments on two collected benchmarks to qualitatively and quantitatively validate the effectiveness of our approach compared with the other state-of-the-art generative models.

Small and accurate heatmap-based face alignment via distillation strategy and cascaded architecture

Despite face alignment has made significant progress, it is still challenging to get a small and accurate face landmark detection model. In this paper, we focus on compressing heatmap-based face alignment algorithms using a novel proposed distillation strategy. We find that the activated areas of heatmaps generated from the well-trained teacher net can capture more local shape information of the facial parts than the ground-truth ones generated from the standard Gaussian distribution. To simultaneously transfer such shape information and correct the mis-predicted heatmaps generated from the teacher model, we first modify the heatmaps of teacher model by replacing the mis-predicted heatmaps with the ground-truth ones as the labels for the student net. To further improve the accuracy of the student net, we investigate the correlation between the extracted features and the predicted heatmaps, and divide the landmarks into two categories: simple and hard. A cascaded architecture is designed which firstly detects the simple points based on the extracted features, and then predicts the hard points resorting to the heatmaps of simple ones. Finally, a face alignment model with 3.64M parameters is obtained, which is about 6x smaller than the cumbersome model, and outperforms the state-of-the-art algorithms on both AFLW2000-3D and 300W-LP. The model and code are released in https://github.com/snow-rgb/Fast-Face-Alignment.

Adversarial Learning for Multiscale Crowd Counting Under Complex Scenes.

In this article, a multiscale generative adversarial network (MS-GAN) is proposed for generating high-quality crowd density maps of arbitrary crowd density scenes. The task of crowd counting has many challenges, such as severe occlusions in extremely dense crowd scenes, perspective distortion, and high visual similarity between the pedestrians and background elements. To address these problems, the proposed MS-GAN combines a multiscale convolutional neural network (generator) and an adversarial network (discriminator) to generate a high-quality density map and accurately estimate the crowd count in complex crowd scenes. The multiscale generator utilizes the fusion features from multiple hierarchical layers to detect people with large-scale variation. The resulting density map produced by the multiscale generator is processed by a discriminator network trained to solve a binary classification task between a poor quality density map and real ground-truth ones. The additional adversarial loss can improve the quality of the density map, which is critical to accurately estimate the crowd counts. The experiments were conducted on multiple datasets with different crowd scenes and densities. The results showed that the proposed method provided better performance compared to current state-of-the-art methods.

Generative dynamic link prediction.

In networks, a link prediction task aims at learning potential relations between nodes to predict unknown potential linkage states. At present, most link prediction methods are used to process static networks. These methods cannot produce good prediction results for dynamic networks. However, for most dynamic networks in the real world, the vertices and links of these networks change over time. Dynamic link prediction (DLP) has attracted more attention as it can better mimic the evolution nature of the networks. Inspired by successful applications of the generative adversarial network in generating fake images, which are comparable with the real ones, we propose a novel generative dynamic link prediction (GDLP) method. Different from other DLP methods, we model the link prediction task as a network generation process. More specifically, GDLP utilizes the historical networks structure information to generate the network snapshot of next time stamp by an end-to-end deep generative model. This model contains a generator and a discriminator. The generator of GDLP is a spatiotemporal prediction model, which is responsible for generating the future networks based on the historical network snapshots, while the discriminator is a classification model to classify the generated networks and the ground-truth ones. With the two-player game training and learning strategy, GDLP is capable of accurate prediction for dynamic networks using the structural and temporal information. Experimental results validate that GDLP significantly outperforms several existing baseline methods on many types of dynamic networks, which improves the effectiveness of dynamic link prediction.

Visual Prediction of Typhoon Clouds With Hierarchical Generative Adversarial Networks

We develop a hierarchical generative adversarial network (HGAN) for generating future typhoon cloud remote sensing images, which enables a visual means to typhoon cloud prediction. The HGAN consists of a global generator and a local discriminator. The global generator aims at producing the future typhoon cloud images as realistic as possible and accordingly reveals the structure and future location of the typhoon clouds. It is constructed in terms of a hierarchical architecture with multiple subnetworks, which capture the overall typhoon variations and favor generating clear future typhoon cloud images. The local discriminator tries its best to distinguish generated typhoon cloud images from ground-truth ones, based on the local patches. The local procedure encourages the discriminator to focus on characterizing the moving typhoon clouds rather than the still background. The global generator and the local discriminator are trained in an adversarial fashion with respect to historical typhoon cloud image sequences. The trained HGAN is capable of producing reliable visual predictions that are not only enabled by the global generator and but also examined by the local discriminator. Experiments validate the effectiveness of the HGAN for typhoon cloud prediction.

Deep face normalization

From angling smiles to duck faces, all kinds of facial expressions can be seen in selfies, portraits, and Internet pictures. These photos are taken from various camera types, and under a vast range of angles and lighting conditions. We present a deep learning framework that can fully normalize unconstrained face images, i.e., remove perspective distortions, relight to an evenly lit environment, and predict a frontal and neutral face. Our method can produce a high resolution image while preserving important facial details and the likeness of the subject, along with the original background. We divide this ill-posed problem into three consecutive normalization steps, each using a different generative adversarial network that acts as an image generator. Perspective distortion removal is performed using a dense flow field predictor. A uniformly illuminated face is obtained using a lighting translation network, and the facial expression is neutralized using a generalized facial expression synthesis framework combined with a regression network based on deep features for facial recognition. We introduce new data representations for conditional inference, as well as training methods for supervised learning to ensure that different expressions of the same person can yield to not only a plausible but also a similar neutral face. We demonstrate our results on a wide range of challenging images collected in the wild. Key applications of our method range from robust image-based 3D avatar creation, portrait manipulation, to facial enhancement and reconstruction tasks for crime investigation. We also found through an extensive user study, that our normalization results can be hardly distinguished from ground truth ones if the person is not familiar.

Adversarial Learning for Constrained Image Splicing Detection and Localization Based on Atrous Convolution

Constrained image splicing detection and localization (CISDL), which investigates two input suspected images and identifies whether one image has suspected regions pasted from the other, is a newly proposed challenging task for image forensics. In this paper, we propose a novel adversarial learning framework to learn a deep matching network for CISDL. Our framework mainly consists of three building blocks. First, a deep matching network based on atrous convolution (DMAC) aims to generate two high-quality candidate masks, which indicate suspected regions of the two input images. In DMAC, atrous convolution is adopted to extract features with rich spatial information, a correlation layer based on a skip architecture is proposed to capture hierarchical features, and atrous spatial pyramid pooling is constructed to localize tampered regions at multiple scales. Second, a detection network is designed to rectify inconsistencies between the two corresponding candidate masks. Finally, a discriminative network drives the DMAC network to produce masks that are hard to distinguish from ground-truth ones. The detection network and the discriminative network collaboratively supervise the training of DMAC in an adversarial way. Besides, a sliding window-based matching strategy is investigated for high-resolution images matching. Extensive experiments, conducted on five groups of datasets, demonstrate the effectiveness of the proposed framework and the superior performance of DMAC.