Image Representation Ability Research Articles

CADS: A Self-supervised Learner via Cross-modal Alignment and Deep Self-distillation for CT Volume Segmentation.

Self-supervised learning (SSL) has long had great success in advancing the field of annotation-efficient learning. However, when applied to CT volume segmentation, most SSL methods suffer from two limitations, including rarely using the information acquired by different imaging modalities and providing supervision only to the bottleneck encoder layer. To address both limitations, we design a pretext task to align the information in each 3D CT volume and the corresponding 2D generated X-ray image and extend self-distillation to deep self-distillation. Thus, we propose a self-supervised learner based on Cross-modal Alignment and Deep Self-distillation (CADS) to improve the encoder's ability to characterize CT volumes. The cross-modal alignment is a more challenging pretext task that forces the encoder to learn better image representation ability. Deep self-distillation provides supervision to not only the bottleneck layer but also shallow layers, thus boosting the abilities of both. Comparative experiments show that, during pre-training, our CADS has lower computational complexity and GPU memory cost than competing SSL methods. Based on the pre-trained encoder, we construct PVT-UNet for 3D CT volume segmentation. Our results on seven downstream tasks indicate that PVT-UNet outperforms state-of-the-art SSL methods like MOCOv3 and DiRA, as well as prevalent medical image segmentation methods like nnUNet and CoTr. Code and pre-trained weight will be available at https://github.com/yeerwen/CADS.

Microsnoop: A generalist tool for microscopy image representation

Accurate profiling of microscopy images from small scale to high throughput is an essential procedure in basic and applied biological research. Here, we present Microsnoop, a novel deep learning-based representation tool trained on large-scale microscopy images using masked self-supervised learning. Microsnoop can process various complex and heterogeneous images, and we classified images into three categories: single-cell, full-field, and batch-experiment images. Our benchmark study on 10 high-quality evaluation datasets, containing over 2,230,000 images, demonstrated Microsnoop's robust and state-of-the-art microscopy image representation ability, surpassing existing generalist and even several custom algorithms. Microsnoop can be integrated with other pipelines to perform tasks such as superresolution histopathology image and multimodal analysis. Furthermore, Microsnoop can be adapted to various hardware and can be easily deployed on local or cloud computing platforms. We will regularly retrain and reevaluate the model using community-contributed data to consistently improve Microsnoop.

Reconstruction-Driven Dynamic Refinement Based Unsupervised Domain Adaptation for Joint Optic Disc and Cup Segmentation.

Glaucoma is one of the leading causes of irreversible blindness. Segmentation of optic disc (OD) and optic cup (OC) on fundus images is a crucial step in glaucoma screening. Although many deep learning models have been constructed for this task, it remains challenging to train an OD/OC segmentation model that could be deployed successfully to different healthcare centers. The difficulties mainly comes from the domain shift issue, i.e., the fundus images collected at these centers usually vary greatly in the tone, contrast, and brightness. To address this issue, in this paper, we propose a novel unsupervised domain adaptation (UDA) method called Reconstruction-driven Dynamic Refinement Network (RDR-Net), where we employ a due-path segmentation backbone for simultaneous edge detection and region prediction and design three modules to alleviate the domain gap. The reconstruction alignment (RA) module uses a variational auto-encoder (VAE) to reconstruct the input image and thus boosts the image representation ability of the network in a self-supervised way. It also uses a style-consistency constraint to force the network to retain more domain-invariant information. The low-level feature refinement (LFR) module employs input-specific dynamic convolutions to suppress the domain-variant information in the obtained low-level features. The prediction-map alignment (PMA) module elaborates the entropy-driven adversarial learning to encourage the network to generate source-like boundaries and regions. We evaluated our RDR-Net against state-of-the-art solutions on four public fundus image datasets. Our results indicate that RDR-Net is superior to competing models in both segmentation performance and generalization ability.

Joint optimization for attention-based generation and recognition of chinese characters using tree position embedding

Despite the growing interest in Chinese character generation, creating a nonexistent character remains an open challenge. Radical-based Chinese character generation is still a novel task while radical-based Chinese character recognition is more technologically advanced. To fully utilize the knowledge of recognition task, we first propose an attention-based generator. The generator chooses the most relevant radical to generate each zone with an attention mechanism. Then, we present a joint optimization approach to training generation-recognition models, which can help the generator and recognizer learn from each other effectively. The joint optimization is implemented via contrastive learning and dual learning. Considering the symmetry of the generation and recognition, contrastive learning aims to strengthen the performance of the encoder of recognizer and the decoder of generator. Since the generation and recognition tasks can form a closed loop, dual learning feeds the output from one to another as input. Based on the feedback signals generated during the two tasks, we can iteratively update the two models until convergence. Finally, as our model ignores the order information of a sequence, we exploit position embedding to extend the image representation ability and propose tree position embedding to represent the positional information for tree structure captions of Chinese characters. The experimental results in printed and nature scenes show that the proposed method improves the quality of the generating images and increases the recognition accuracy for Chinese characters.

MsIFT: Multi-Source Image Fusion Transformer

Multi-source image fusion is very important for improving image representation ability since its essence relies on the complementarity between multi-source information. However, feature-level image fusion methods based on the convolution neural network are impacted by the spatial misalignment between image pairs, which leads to the semantic bias in merging features and destroys the representation ability of the region-of-interests. In this paper, a novel multi-source image fusion transformer (MsIFT) is proposed. Due to the inherent global attention mechanism of the transformer, the MsIFT has non-local fusion receptive fields, and it is more robust to spatial misalignment. Furthermore, multiple classification-based downstream tasks (e.g., pixel-wise classification, image-wise classification and semantic segmentation) are unified in the proposed MsIFT framework, and the fusion module architecture is shared by different tasks. The MsIFT achieved state-of-the-art performances on the image-wise classification dataset VAIS, semantic segmentation dataset SpaceNet 6 and pixel-wise classification dataset GRSS-DFC-2013. The code and trained model are being released upon the publication of the work.

Learning From Ambiguous Labels for Lung Nodule Malignancy Prediction.

Lung nodule malignancy prediction is an essential step in the early diagnosis of lung cancer. Besides the difficulties commonly discussed, the challenges of this task also come from the ambiguous labels provided by annotators, since deep learning models have in some cases been found to reproduce or amplify human biases. In this paper, we propose a multi-view 'divide-and-rule' (MV-DAR) model to learn from both reliable and ambiguous annotations for lung nodule malignancy prediction on chest CT scans. According to the consistency and reliability of their annotations, we divide nodules into three sets: a consistent and reliable set (CR-Set), an inconsistent set (IC-Set), and a low reliable set (LR-Set). The nodule in IC-Set is annotated by multiple radiologists inconsistently, and the nodule in LR-Set is annotated by only one radiologist. Although ambiguous, inconsistent labels tell which label(s) is consistently excluded by all annotators, and the unreliable labels of a cohort of nodules are largely correct from the statistical point of view. Hence, both IC-Set and LR-Set can be used to facilitate the training of MV-DAR. Our MV-DAR contains three DAR models to characterize a lung nodule from three orthographic views and is trained following a two-stage procedure. Each DAR consists of three networks with the same architecture, including a prediction network (Prd-Net), a counterfactual network (CF-Net), and a low reliable network (LR-Net), which are trained on CR-Set, IC-Set, and LR-Set respectively in the pretraining phase. In the fine-tuning phase, the image representation ability learned by CF-Net and LR-Net is transferred to Prd-Net by negative-attention module (NA-Module) and consistent-attention module (CA-Module), aiming to boost the prediction ability of Prd-Net. The MV-DAR model has been evaluated on the LIDC-IDRI dataset and LUNGx dataset. Our results indicate not only the effectiveness of the MV-DAR in learning from ambiguous labels but also its superiority over present noisy label-learning models in lung nodule malignancy prediction.

Image Analysis by Fractional-Order Gaussian-Hermite Moments.

Moments and moment invariants are effective feature descriptors. They have widespread applications in the field of image processing. The recent researches show that fractional-order moments have notable image representation ability. Hermite polynomials are defined over the interval from negative infinity to positive one. Such unboundedness prevents us from developing fractional-order Gaussian-Hermite moments via the existing ideas or approaches. In this paper, we propose fractional-order Gaussian-Hermite moments by forcing the definition domain of Hermite polynomials to be a bounded interval, meanwhile, resorting to a value-decreasing standard deviation to maintain the orthogonality. Moreover, we successfully develop contrast, translation and rotation invariants from the proposed moments based on the inherent properties of Hermite polynomials. The reconstructions of different types of images demonstrate that the proposed moments have more superior image representation ability to the most existing popular orthogonal moments. Besides, the salient performance in invariant image recognition, noise robustness and region-of-interest feature extraction reflect that these moments and their invariants possess the stronger discrimination power and the better noise robustness in comparison with the existing orthogonal moments. Furthermore, both complexity analysis and time consumption indicate that the proposed moments and their invariants are easy to implement, they are suitable for practical engineering applications.

전래동화를 활용한 미술 활동이 유아의 정서지능과 그림표상능력에 미치는 영향

본 연구의 목적은 전래동화를 활용한 미술 활동이 유아의 정서지능과 그림표상능력에 미치는 영향을 살펴보는 것이다. 연구 대상은 광주광역시에 소재한 B 어린이집에 재원 중인 만 5세 유아 15명(실험집단)과 G 어린이집에 재원 중인 만 5세 유아 15명(비교집단)이었다. 실험집단은 전래동화를 활용한 미술 활동에, 비교집단은 누리과정의 생활주제에 따른 미술 활동에 참여하였다. 연구결과는 정서지능의 4개 하위영역과 그림표상능력에서 실험집단 점수가 비교집단보다 유의하게 향상한 것으로 나타났다. 전래동화를 활용한 미술 활동이 유아의 정서지능과 그림표상능력 향상에 긍정적 영향을 미치는 것이다. 등장인물의 성격이 분명하고 환상적이며 흥미로운 내용을 지닌 전래동화를 미술 활동에 활용함으로써 본 연구는 유아의 정서지능과 그림표상능력을 효과적으로 증진하는 교수-학습 활동을 보여준다. 결국 본 연구는 유아의 정서지능과 그림표상능력의 증진에 전래동화를 활용한 미술 활동이 효과적임을 입증하는 데 그 의의가 있다.

Fusing Multilevel Deep Features for Fabric Defect Detection Based NTV-RPCA

Fabric defect detection plays an important role in automated inspection and quality control in textile manufacturing. As the fabric images have complex and diverse textures and defects, traditional detection methods show a poor adaptability and low detection accuracy. Robust principal component analysis (RPCA) model that can be used to separate the image into object and background have proven applicable in fabric defect detection. However, how to represent texture feature of the fabric image more effectively is still problematic in this kind of method. In addition, the use of the traditional RPCA may result in low accuracy and more noises in sparse part. In this article, a novel fabric defect detection method based on multilevel deep features fusion and non-convex total variation regularized RPCA (NTV-RPCA) is proposed. Firstly, the image representation ability is well enhanced through multilevel deep features extracted by a convolutional neural network. Then, the non-convex total variation regularized RPCA is proposed in which total variation constraint significantly reduces the noises in sparse part and non-convex solution is more approximate to the authentic one. Next, multilevel saliency maps generated by the sparse matrixes are fused via RPCA to produce a more reliable detection result. Finally, the defect region is located by segmenting the fused saliency map via a threshold segmentation algorithm. Qualitative and quantitative experiments conducted on two public fabric image databases demonstrate that the proposed method improves the adaptability and detection accuracy comparing to the state-of-the-arts.

Semi-supervised adversarial model for benign-malignant lung nodule classification on chest CT.

Classification of benign-malignant lung nodules on chest CT is the most critical step in the early detection of lung cancer and prolongation of patient survival. Despite their success in image classification, deep convolutional neural networks (DCNNs) always require a large number of labeled training data, which are not available for most medical image analysis applications due to the work required in image acquisition and particularly image annotation. In this paper, we propose a semi-supervised adversarial classification (SSAC) model that can be trained by using both labeled and unlabeled data for benign-malignant lung nodule classification. This model consists of an adversarial autoencoder-based unsupervised reconstruction network R, a supervised classification network C, and learnable transition layers that enable the adaption of the image representation ability learned by R to C. The SSAC model has been extended to the multi-view knowledge-based collaborative learning, aiming to employ three SSACs to characterize each nodule's overall appearance, heterogeneity in shape and texture, respectively, and to perform such characterization on nine planar views. The MK-SSAC model has been evaluated on the benchmark LIDC-IDRI dataset and achieves an accuracy of 92.53% and an AUC of 95.81%, which are superior to the performance of other lung nodule classification and semi-supervised learning approaches.

ConvNet and LSH-Based Visual Localization Using Localized Sequence Matching.

Convolutional Network (ConvNet), with its strong image representation ability, has achieved significant progress in the computer vision and robotic fields. In this paper, we propose a visual localization approach based on place recognition that combines the powerful ConvNet features and localized image sequence matching. The image distance matrix is constructed based on the cosine distance of extracted ConvNet features, and then a sequence search technique is applied on this distance matrix for the final visual recognition. To speed up the computational efficiency, the locality sensitive hashing (LSH) method is applied to achieve real-time performances with minimal accuracy degradation. We present extensive experiments on four real world data sets to evaluate each of the specific challenges in visual recognition. A comprehensive performance comparison of different ConvNet layers (each defining a level of features) considering both appearance and illumination changes is conducted. Compared with the traditional approaches based on hand-crafted features and single image matching, the proposed method shows good performances even in the presence of appearance and illumination changes.

A Joint Convolutional Neural Networks and Context Transfer for Street Scenes Labeling

Street scene understanding is an essential task for autonomous driving. One important step towards this direction is scene labeling, which annotates each pixel in the images with a correct class label. Although many approaches have been developed, there are still some weak points. Firstly, many methods are based on the hand-crafted features whose image representation ability is limited. Secondly, they can not label foreground objects accurately due to the dataset bias. Thirdly, in the refinement stage, the traditional Markov Random Filed (MRF) inference is prone to over smoothness. For improving the above problems, this paper proposes a joint method of priori convolutional neural networks at superpixel level (called as ``priori s-CNNs'') and soft restricted context transfer. Our contributions are threefold: (1) A priori s-CNNs model that learns priori location information at superpixel level is proposed to describe various objects discriminatingly; (2) A hierarchical data augmentation method is presented to alleviate dataset bias in the priori s-CNNs training stage, which improves foreground objects labeling significantly; (3) A soft restricted MRF energy function is defined to improve the priori s-CNNs model's labeling performance and reduce the over smoothness at the same time. The proposed approach is verified on CamVid dataset (11 classes) and SIFT Flow Street dataset (16 classes) and achieves competitive performance.

Image analysis by Gaussian–Hermite moments

Orthogonal moments are powerful tools in pattern recognition and image processing applications. In this paper, the Gaussian–Hermite moments based on a set of orthonormal weighted Hermite polynomials are extensively studied. The rotation and translation invariants of Gaussian–Hermite moments are derived algebraically. It is proved that the construction forms of geometric moment invariants are valid for building the Gaussian–Hermite moment invariants. The paper also discusses the computational aspects of Gaussian–Hermite moment, including the recurrence relation and symmetrical property. Just as the other orthogonal moments, an image can be easily reconstructed from its Gaussian–Hermite moments thanks to the orthogonality of the basis functions. Some reconstruction tests with binary and gray-level images (without and with noise) were performed and the obtained results show that the reconstruction quality from Gaussian–Hermite moments is better than that from known Legendre, discrete Tchebichef and Krawtchouk moments. This means Gaussian–Hermite moment has higher image representation ability. The peculiarity of image reconstruction algorithm from Gaussian–Hermite moments is also discussed in the paper. The paper offers an example of classification using Gaussian–Hermite moment invariants as pattern feature and the result demonstrates that Gaussian–Hermite moment invariants perform significantly better than Hu's moment invariants under both noise-free and noisy conditions.

Transform based spatio-temporal descriptors for human action recognition

Classic transformation methods have been widely and efficiently used in image processing areas, such as image de-noising, image segmentation, feature detection, and compression. Based on their compact signal and image representation ability, we apply the transform based techniques on the video recognition area to extract discriminative information from each given video sequence, and use the transformed coefficients as descriptors for representing and recognizing human actions in video sequences. We validate our proposed methods on the KTH and the Hollywood datasets, which have been extensively studied by a lot of researchers. The proposed descriptors, especially the wavelet transform based descriptor, yield promising results on action recognition.

Robust and Efficient Fourier–Mellin Transform Approximations for Gray-Level Image Reconstruction and Complete Invariant Description

This paper addresses the gray-level image representation ability of the Fourier–Mellin transform (FMT) for pattern recognition, reconstruction, and image database retrieval. The main practical difficulty of the FMT lies in the accuracy and efficiency of its numerical approximation and we propose three estimations of its analytical extension. A comparison of these approximations is performed from discrete and finite-extent sets of Fourier–Mellin harmonics by means of experiments in: (i) image reconstruction via both visual inspection and the computation of a reconstruction error; and (ii) pattern recognition and discrimination by using a complete and convergent set of features invariant under planar similarities.Experimental results on real gray-level images show that it is possible to recover an image to within a specified degree of accuracy and to classify objects reliably even when a large set of descriptors is used. Finally, an example will be given, which illustrates both theoretical and numerical results in the context of content-based image retrieval.

On image analysis by the methods of moments

Various types of moments have been used to recognize image patterns in a number of applications. A number of moments are evaluated and some fundamental questions are addressed, such as image-representation ability, noise sensitivity, and information redundancy. Moments considered include regular moments, Legendre moments, Zernike moments, pseudo-Zernike moments, rotational moments, and complex moments. Properties of these moments are examined in detail and the interrelationships among them are discussed. Both theoretical and experimental results are presented. >