Features Of Natural Images Research Articles

MTC-CSNet: Marrying Transformer and Convolution for Image Compressed Sensing.

Image compressed sensing (ICS) has been extensively applied in various imaging domains due to its capability to sample and reconstruct images at subNyquist sampling rates. The current predominant approaches in ICS, specifically pure convolutional networks (ConvNets)-based ICS methods, have demonstrated their effectiveness in capturing local features for image recovery. Simultaneously, the Transformer architecture has gained significant attention due to its capability to model global correlations among image features. Motivated by these insights, we propose a novel hybrid network for ICS, named MTC-CSNet, which effectively combines the strengths of both ConvNets and Transformer architectures in capturing local and global image features to achieve high-quality image recovery. Particularly, MTC-CSNet is a dual-path framework that consists of a ConvNets-based recovery branch and a Transformer-based recovery branch. Along the ConvNets-based recovery branch, we design a lightweight scheme to capture the local features in natural images. Meanwhile, we implement a Transformer-based recovery branch to iteratively model the global dependencies among image patches. Ultimately, the ConvNets-based and Transformer-based recovery branches collaborate through a bridging unit, facilitating the adaptive transmission and fusion of informative features for image reconstruction. Extensive experimental results demonstrate that our proposed MTC-CSNet surpasses the state-of-the-art methods on various public datasets. The code and models are publicly available at MTC-CSNet.

Large-scale calcium imaging reveals a systematic V4 map for encoding natural scenes

Biological visual systems have evolved to process natural scenes. A full understanding of visual cortical functions requires a comprehensive characterization of how neuronal populations in each visual area encode natural scenes. Here, we utilized widefield calcium imaging to record V4 cortical response to tens of thousands of natural images in male macaques. Using this large dataset, we developed a deep-learning digital twin of V4 that allowed us to map the natural image preferences of the neural population at 100-µm scale. This detailed map revealed a diverse set of functional domains in V4, each encoding distinct natural image features. We validated these model predictions using additional widefield imaging and single-cell resolution two-photon imaging. Feature attribution analysis revealed that these domains lie along a continuum from preferring spatially localized shape features to preferring spatially dispersed surface features. These results provide insights into the organizing principles that govern natural scene encoding in V4.

Occipital-temporal cortical tuning to semantic and affective features of natural images predicts associated behavioral responses

In everyday life, people need to respond appropriately to many types of emotional stimuli. Here, we investigate whether human occipital-temporal cortex (OTC) shows co-representation of the semantic category and affective content of visual stimuli. We also explore whether OTC transformation of semantic and affective features extracts information of value for guiding behavior. Participants viewed 1620 emotional natural images while functional magnetic resonance imaging data were acquired. Using voxel-wise modeling we show widespread tuning to semantic and affective image features across OTC. The top three principal components underlying OTC voxel-wise responses to image features encoded stimulus animacy, stimulus arousal and interactions of animacy with stimulus valence and arousal. At low to moderate dimensionality, OTC tuning patterns predicted behavioral responses linked to each image better than regressors directly based on image features. This is consistent with OTC representing stimulus semantic category and affective content in a manner suited to guiding behavior.

Representation of Natural Contours by a Neural Population in Monkey V4.

The cortical visual area, V4, has been considered to code contours that contribute to the intermediate-level representation of objects. The neural responses to the complex contour features intrinsic to natural contours are expected to clarify the essence of the representation. To approach the cortical coding of natural contours, we investigated the simultaneous coding of multiple contour features in monkey (Macaca fuscata) V4 neurons and their population-level representation. A substantial number of neurons showed significant tuning for two or more features such as curvature and closure, indicating that a substantial number of V4 neurons simultaneously code multiple contour features. A large portion of the neurons responded vigorously to acutely curved contours that surrounded the center of classical receptive field, suggesting that V4 neurons tend to code prominent features of object contours. The analysis of mutual information (MI) between the neural responses and each contour feature showed that most neurons exhibited similar magnitudes for each type of MI, indicating that many neurons showing the responses depended on multiple contour features. We next examined the population-level representation by using multidimensional scaling analysis. The neural preferences to the multiple contour features and that to natural stimuli compared with silhouette stimuli increased along with the primary and secondary axes, respectively, indicating the contribution of the multiple contour features and surface textures in the population responses. Our analyses suggested that V4 neurons simultaneously code multiple contour features in natural images and represent contour and surface properties in population.

Mineral identification based on natural feature-oriented image processing and multi-label image classification

Artificial intelligence (AI) technology has significant potential in Earth sciences, particularly in mineral identification for industrial exploration, geological mapping, and archaeological research. However, traditional methods are time-consuming, expensive, and complex. And existing mineral identification methods based on mineral photos face several critical challenges, including lack of consideration for natural image features captured in real environments, limitations of single-label classification which does not align with multi-mineral occurrences in nature, and growing computational complexity as the number of identifiable mineral labels increases. Therefore, this paper proposes an efficient mineral identification model based on multi-label image classification, focusing on natural environmental features. First, realistic feature datasets are created by simulating mineral photos in real environments. Then, the model uses the query-label (Query2Label) framework, with MaxViT-T (Multi-Axis Vision Transformer-Tiny) as the feature extraction network and the asymmetric loss function. Knowledge distillation is employed to improve identification accuracy while reducing computational complexity. The proposed model achieves an impressive average identification accuracy of 84.74% on a dataset of 495,756 mineral photos, surpassing existing models like ResNet-101, ML-GCN (Multi-Label Graph Convolutional Network), and SRN (Spatial Regularization Net). It maintains a lower parameter count and computational complexity. In the end, ablation experiments demonstrate the effectiveness of each optimization scheme.

Multi-source adversarial transfer learning for ultrasound image segmentation with limited similarity

Lesion segmentation of ultrasound medical images based on deep learning techniques is a widely used method for diagnosing diseases. Although there is a large amount of ultrasound image data in medical centers and other places, labeled ultrasound datasets are a scarce resource, and it is likely that no datasets are available for new tissues/organs. Transfer learning provides the possibility to solve this problem, but there are too many features in natural images that are not related to the target domain. As a source domain, redundant features that are not conducive to the task will be extracted. Migration between ultrasound images can avoid this problem, but there are few types of public datasets, and it is difficult to find sufficiently similar source domains. Compared with natural images, ultrasound images have less information, and there are fewer transferable features between different ultrasound images, which may cause negative transfer. To this end, a multi-source adversarial transfer learning network for ultrasound image segmentation is proposed. Specifically, to address the lack of annotations, the idea of adversarial transfer learning is used to adaptively extract common features between a certain pair of source and target domains, which provides the possibility to utilize unlabeled ultrasound data. To alleviate the lack of knowledge in a single source domain, multi-source transfer learning is adopted to fuse knowledge from multiple source domains. In order to ensure the effectiveness of the fusion and maximize the use of precious data, a multi-source domain independent strategy is also proposed to improve the estimation of the target domain data distribution, which further increases the learning ability of the multi-source adversarial migration learning network in multiple domains. The effectiveness of multi-source adversarial transfer learning is demonstrated through experiments on three datasets of ultrasound image datasets.

Contour detection network for zero-shot sketch-based image retrieval

Zero-shot sketch-based image retrieval (ZS-SBIR) is a challenging task that involves searching natural images related to a given hand-drawn sketch under the zero-shot scene. The previous approach projected image and sketch features into a low-dimensional common space for retrieval, and used semantic features to transfer the knowledge of seen to unseen classes. However, it is not effective enough to align multimodal features when projecting them into a common space, since the styles and contents of sketches and natural images are different and they are not one-to-one correspondence. To solve this problem, we propose a novel three-branch joint training network with contour detection network (called CDNNet) for the ZS-SBIR task, which uses contour maps as a bridge to align sketches and natural images to alleviate the domain gap. Specifically, we use semantic metrics to constrain the relationship between contour images and natural images and between contour images and sketches, so that natural image and sketch features can be aligned in the common space. Meanwhile, we further employ second-order attention to capture target subject information to increase the performance of retrieval descriptors. In addition, we use a teacher model and word embedding method to transfer the knowledge of the seen to the unseen classes. Extensive experiments on two large-scale datasets demonstrate that our proposed approach outperforms state-of-the-art CNN-based models: it improves by 2.6% on the Sketchy and 1.2% on TU-Berlin datasets in terms of mAP.

Towed Streamer-Based Simultaneous Source Separation by Contourlet Transform

Simultaneous towed-streamer marine acquisition has the advantages of reducing the total time requirements and costs of surveys. However, the seismic records obtained are blended seismic data, so the successful deblending of such data is the key to this method. In this paper, we propose an effective deblending method with a new thresholding operator based on the shaping regularization framework in the contourlet domain. The new thresholding operator consists of an adaptive Bayesian threshold and a new thresholding function. Because of its multiresolution, locality, and directionality properties, the contourlet transform can effectively capture geometrical structures, which are the main features in natural images. To make the traditional Bayesian threshold adaptive in the contourlet domain, we propose a scale adjustment factor, a direction adjustment factor, and an attenuation factor to modify the threshold, and we also adopt local adaptive elliptic windows to estimate the standard deviations of useful signals; eventually, we obtain an adaptive Bayesian threshold. Furthermore, the new thresholding function can overcome the shortcomings of the existing soft and hard thresholding functions. Experimental results demonstrate that our method can effectively separate blended data.

Slow Feature Extraction Algorithm Based on Visual Selection Consistency Continuity and Its Application

The conventional slow feature analysis (SFA) algorithm has no support of computational theory of vision for primates, nor does it have the ability to learn the global features with visual selection consistency continuity. And what is more, the algorithm is highly complex. Based on this, Slow Feature Extraction Algorithm Based on Visual selection consistency continuity and Its Application was proposed. Inspired by the visual selection consistency continuity theory for primates, this paper replaced the principal component analysis (PCA) method of the conventional SFA algorithm with the myTICA method, extracted the Gabor basis functions of natural images, initialized the basis function family; it used the feature basis expansion algorithm based on visual selection consistency continuity (the VSCC_FBEA algorithm) to replace the polynomial expansion method in the original SFA algorithm to generates the Gabor basis functions of features with long and short-term visual selectivity in the family of basis functions, which solved the drawbacks of the polynomial prediction algorithm; it also designed the Lipschitz consistency constraint, and proposed the Lipschitz-Orthogonal-Pruning-Method (LOPM algorithm) to optimize the basis function family into an over-complete family of basis functions. In addition, this paper used the feature expression method based on visual invariance theory (visual invariance theory -FEM) to establish the set of features of natural images with visual selection consistency continuity. Subsequently, it adopted three error evaluation methods and mySFA classification method to evaluate the proposed algorithm. According to the experimental results, the proposed algorithm showed good prediction performance with respect to the LSVRC2012 data set; compared with the SFA, GSFA, TICA, myICA and mySFA algorithms, the proposed algorithm is correct and feasible; when the classification threshold of the algorithm was set at 8.0, the recognition rate of the proposed algorithm reached 99.66%, and neither of the false recognition rate and the false rejection rate was higher than 0.33%. The proposed algorithm has good performance in prediction and classification, and also shows good anti-noise capacity under limited noise conditions.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings.

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Decontextualized learning for interpretable hierarchical representations of visual patterns

SummaryApart from discriminative modeling, the application of deep convolutional neural networks to basic research utilizing natural imaging data faces unique hurdles. Here, we present decontextualized hierarchical representation learning (DHRL), designed specifically to overcome these limitations. DHRL enables the broader use of small datasets, which are typical in most studies. It also captures spatial relationships between features, provides novel tools for investigating latent variables, and achieves state-of-the-art disentanglement scores on small datasets. DHRL is enabled by a novel preprocessing technique inspired by generative model chaining and an improved ladder network architecture and regularization scheme. More than an analytical tool, DHRL enables novel capabilities for virtual experiments performed directly on a latent representation, which may transform the way we perform investigations of natural image features, directly integrating analytical, empirical, and theoretical approaches.

Learning Rotation Domain Deep Mutual Information Using Convolutional LSTM for Unsupervised PolSAR Image Classification

Deep learning can archive state-of-the-art performance in polarimetric synthetic aperture radar (PolSAR) image classification with plenty of labeled data. However, obtaining large number of accurately labeled samples of PolSAR data is very hard, which limits the practical use of deep learning. Therefore, unsupervised PolSAR image classification is worthy of further investigation that is based on deep learning. Inspired by the superior performance of deep mutual information in natural image feature learning and clustering, an end-to-end Convolutional Long Short Term Memory (ConvLSTM) network is used in order to learn the deep mutual information of polarimetric coherent matrices in the rotation domain with different polarimetric orientation angles (POAs) for unsupervised PolSAR image classification. First, for each pixel, paired “POA-spatio” samples are generated from the polarimetric coherent matrices with different POAs. Second, a special designed ConvLSTM network, along with deep mutual information losses, is used in order to learn the discriminative deep mutual information feature representation of the paired data. Finally, the classification results can be output directly from the trained network model. The proposed method is trained in an end-to-end manner and does not have cumbersome pipelines. Experiments on four real PolSAR datasets show that the performance of proposed method surpasses some state-of-the-art deep learning unsupervised classification methods.

Deep Natural Image Reconstruction from Human Brain Activity Based on Conditional Progressively Growing Generative Adversarial Networks.

Brain decoding based on functional magnetic resonance imaging has recently enabled the identification of visual perception and mental states. However, due to the limitations of sample size and the lack of an effective reconstruction model, accurate reconstruction of natural images is still a major challenge. The current, rapid development of deep learning models provides the possibility of overcoming these obstacles. Here, we propose a deep learning-based framework that includes a latent feature extractor, a latent feature decoder, and a natural image generator, to achieve the accurate reconstruction of natural images from brain activity. The latent feature extractor is used to extract the latent features of natural images. The latent feature decoder predicts the latent features of natural images based on the response signals from the higher visual cortex. The natural image generator is applied to generate reconstructed images from the predicted latent features of natural images and the response signals from the visual cortex. Quantitative and qualitative evaluations were conducted with test images. The results showed that the reconstructed image achieved comparable, accurate reproduction of the presented image in both high-level semantic category information and low-level pixel information. The framework we propose shows promise for decoding the brain activity.

The Comparative Study on Underlying Features of PainingTang-ka Images and Natural Images

Our purpose was to find the difference between the color features and texture features of natural images and painted Tang-ka images. Taking natural images as a reference, painted one were analyzed in HSV color space. The natural image and painted Thang-ga texture features were extracted, using the Tamura’s method. The data showed that the natural image’s coarse was larger than the painted Tang-ka, and painted Tang-ka were more saturated than natural images. Compared with natural images, the painted Tang-ka’s directonality has higher value. By contrast, the red Tang-ka had a higher value than the golden one.

Correlating the natural color of tropical fruit juice with its pH

AbstractColor analysis is a critical approach used to analyze the quality properties of fruits, particularly in tracing the ripening stage through the peel color of the fruits. In this study, the color features of natural and untreated tropical fruit juice images (ie, B10 Averrhoa carambola L. and Sala mango) were correlated with the pH of the juice samples. The CIELAB color characteristics of the fruits showed that the mango juice had darker features than the carambola juice. The cause may be due to the higher soluble solid content and denser yellow and red pigments in the mango juice. For carambola juice, multiple linear regression (MLR) performed on red‐blue (RB) components generated the best prediction of pH with the highest coefficient of determination and lowest root mean square of error. For mango juice, MLR performed on normalized red‐green‐blue (RGB) components generated the best prediction of pH.

Blind motion deblurring with cycle generative adversarial networks

Blind motion deblurring is one of the most basic and challenging problems in image processing and computer vision. It aims to recover a sharp image from its blurred version knowing nothing about the blur process. Many existing methods use Maximum A Posteriori (MAP) or Expectation Maximization (EM) frameworks to deal with this kind of problems, but they cannot handle well the figh frequency features of natural images. Most recently, deep neural networks have been emerging as a powerful tool for image deblurring. In this paper, we prove that encoder-decoder architecture gives better results for image deblurring tasks. In addition, we propose a novel end-to-end learning model which refines generative adversarial network by many novel training strategies so as to tackle the problem of deblurring. Experimental results show that our model can capture high frequency features well, and the results on benchmark dataset show that proposed model achieves the competitive performance.

Unambiguous Scene Text Segmentation with Referring Expression Comprehension.

Text instance provides valuable information for the understanding and interpretation of natural scenes. The rich, precise high-level semantics embodied in the text could be beneficial for understanding the world around us, and empower a wide range of real-world applications. While most recent visual phrase grounding approaches focus on general objects, this paper explores extracting designated texts and predicting unambiguous scene text segmentation mask, i.e. scene text segmentation from natural language descriptions (referring expressions) like orange text on a little boy in black swinging a bat. The solution of this novel problem enables accurate segmentation of scene text instances from the complex background. In our proposed framework, a unified deep network jointly models visual and linguistic information by encoding both region-level and pixel-level visual features of natural scene images into spatial feature maps, and then decode them into saliency response map of text instances. To conduct quantitative evaluations, we establish a new scene text referring expression segmentation dataset: COCO-CharRef. Experimental results demonstrate the effectiveness of the proposed framework on the text instance segmentation task. By combining image-based visual features with language-based textual explanations, our framework outperforms baselines that are derived from state-of-the-art text localization and natural language object retrieval methods on COCO-CharRef dataset.

On the Sparse Structure of Natural Sounds and Natural Images: Similarities, Differences, and Implications for Neural Coding.

Sparse coding models of natural images and sounds have been able to predict several response properties of neurons in the visual and auditory systems. While the success of these models suggests that the structure they capture is universal across domains to some degree, it is not yet clear which aspects of this structure are universal and which vary across sensory modalities. To address this, we fit complete and highly overcomplete sparse coding models to natural images and spectrograms of speech and report on differences in the statistics learned by these models. We find several types of sparse features in natural images, which all appear in similar, approximately Laplace distributions, whereas the many types of sparse features in speech exhibit a broad range of sparse distributions, many of which are highly asymmetric. Moreover, individual sparse coding units tend to exhibit higher lifetime sparseness for overcomplete models trained on images compared to those trained on speech. Conversely, population sparseness tends to be greater for these networks trained on speech compared with sparse coding models of natural images. To illustrate the relevance of these findings to neural coding, we studied how they impact a biologically plausible sparse coding network's representations in each sensory modality. In particular, a sparse coding network with synaptically local plasticity rules learns different sparse features from speech data than are found by more conventional sparse coding algorithms, but the learned features are qualitatively the same for these models when trained on natural images.

Effective Airplane Detection in Remote Sensing Images Based on Multilayer Feature Fusion and Improved Nonmaximal Suppression Algorithm

Aiming at the problem of insufficient representation ability of weak and small objects and overlapping detection boxes in airplane object detection, an effective airplane detection method in remote sensing images based on multilayer feature fusion and an improved nonmaximal suppression algorithm is proposed. Firstly, based on the common low-level visual features of natural images and airport remote sensing images, region-based convolutional neural networks are chosen to conduct transfer learning for airplane images using a limited amount of data. Then, the L2 norm normalization, feature connection, scale scaling, and feature dimension reduction are introduced to achieve effective fusion of low- and high-level features. Finally, a nonmaximal suppression method based on a soft decision function is proposed to solve the overlap problem of detection boxes. The experimental results show that the proposed method can effectively improve the representation ability of weak and small objects, as well as quickly and accurately detect airplane objects in the airport area.

Optimized Deep Convolutional Neural Networks for Identification of Macular Diseases from Optical Coherence Tomography Images

Finetuning pre-trained deep neural networks (DNN) delicately designed for large-scale natural images may not be suitable for medical images due to the intrinsic difference between the datasets. We propose a strategy to modify DNNs, which improves their performance on retinal optical coherence tomography (OCT) images. Deep features of pre-trained DNN are high-level features of natural images. These features harm the training of transfer learning. Our strategy is to remove some deep convolutional layers of the state-of-the-art pre-trained networks: GoogLeNet, ResNet and DenseNet. We try to find the optimized deep neural networks on small-scale and large-scale OCT datasets, respectively, in our experiments. Results show that optimized deep neural networks not only reduce computational burden, but also improve classification accuracy.