Shared Feature Representation Research Articles

An Efficient Deep Unsupervised Domain Adaptation for Unknown Malware Detection

As an innovative way of communicating information, the Internet has become an indispensable part of our lives. However, it also facilitates a more widespread attack of malware. With the assistance of modern cryptanalysis, emerging malware having symmetric properties, such as encryption and decryption, pack and unpack, presents new challenges to effective malware detection. Currently, numerous malware detection approaches are based on supervised learning. The biggest challenge is that the existing systems rely on a large amount of labeled data, which is usually difficult to gain. Moreover, since the newly emerging malware has a different data distribution from the original training samples, the detection performance of these systems will degrade along with the emergence of new malware. To solve these problems, we propose an Unsupervised Domain Adaptation (UDA)-based malware detection method by jointly aligning the distribution of known and unknown malware. Firstly, the distribution divergence between the source and target domain is minimized with the help of symmetric adversarial learning to learn shared feature representations. Secondly, to further obtain semantic information of unlabeled target domain data, this paper reduces the class-level distribution divergence by aligning the class center of labeled source and pseudo-labeled target domain data. Finally, we mainly use a residual network with a self-attention mechanism to extract more accurate feature information. A series of experiments are performed on two public datasets. Experimental results illustrate that the proposed approach outperforms the existing detection methods with an accuracy of 95.63% and 95.04% in detecting unknown malware on two datasets, respectively.

Cross-modal distillation for RGB-depth person re-identification

Person re-identification is a key challenge for surveillance across multiple sensors. Prompted by the advent of powerful deep learning models for visual recognition, and inexpensive RGB-D cameras and sensor-rich mobile robotic platforms, e.g. self-driving vehicles, we investigate the relatively unexplored problem of cross-modal re-identification of persons between RGB (color) and depth images. The considerable divergence in data distributions across different sensor modalities introduces additional challenges to the typical difficulties like distinct viewpoints, occlusions, and pose and illumination variation. While some work has investigated re-identification across RGB and infrared, we take inspiration from successes in transfer learning from RGB to depth in object detection tasks. Our main contribution is a novel method for cross-modal distillation for robust person re-identification, which learns a shared feature representation space of person’s appearance in both RGB and depth images. In addition, we propose a cross-modal attention mechanism where the gating signal from one modality can dynamically activate the most discriminant CNN filters of the other modality. The proposed distillation method is compared to conventional and deep learning approaches proposed for other cross-domain re-identification tasks. Results obtained on the public BIWI and RobotPKU datasets indicate that the proposed method can significantly outperform the state-of-the-art approaches by up to 16.1% in mean Average Precision (mAP), demonstrating the benefit of the distillation paradigm. The experimental results also indicate that using cross-modal attention allows to improve recognition accuracy considerably with respect to the proposed distillation method and relevant state-of-the-art approaches.11Code: https://github.com/frhf/cross-modal-distillation-reidentification.

Deep Covariance Alignment for Domain Adaptive Remote Sensing Image Segmentation

Unsupervised domain adaptive (UDA) image segmentation has recently gained increasing attention, aiming to improve the generalization capability for transferring knowledge from the source domain to the target domain. However, in high spatial resolution remote sensing image (RSI), the same category from different domains (e.g., urban and rural) can appear to be totally different with extremely inconsistent distributions, which heavily limits the UDA accuracy. To address this problem, in this article, we propose a novel deep covariance alignment (DCA) model for UDA RSI segmentation. The DCA can explicitly align category features to learn shared domain-invariant discriminative feature representations, which enhance the ability of model generalization. Specifically, a category feature pooling (CFP) module is first used to extract category features by combining coarse outputs and deep features. Then, we leverage a novel covariance regularization (CR) to enforce the intracategory features to be closer and the intercategory features to be further separate. Compared with the existing category alignment methods, our CR aims to regularize the correlation between different dimensions of the features, and thus performs more robustly when dealing with divergent category features of imbalanced and inconsistent distributions. Finally, we propose a stagewise procedure to train the DCA to alleviate error accumulation. Experiments on both rural-to-urban and urban-to-rural scenarios of the LoveDA dataset demonstrate the superiority of our proposed DCA over other state-of-the-art UDA segmentation methods. Code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/Luffy03/DCA</uri> .

Deep Transfer Network With Adaptive Joint Distribution Adaptation: A New Process Fault Diagnosis Model

Distribution discrepancy between training data and testing data caused by varying working conditions limits the wide applications of deep learning-based methods for process fault diagnosis. Generally, this issue is addressed by transfer learning (TL) effectively. However, previous works on TL mainly focus on aligning the marginal distribution only or ignoring the different impacts of the marginal and conditional distributions of the data. Thus, it remains challenging to reduce domain shifts by considering marginal and conditional distributions adaptatively and simultaneously. In this article, a novel deep transfer network (DTN) with adaptive joint distribution adaptation (AJDA) is proposed to solve the problem of process fault diagnosis under varying working conditions. First, an adaptive joint distribution module is proposed to implement domain adaptation both in feature space and label space. AJDA not only aligns the marginal and conditional distribution simultaneously but also quantifies the importance of the two distributions. Moreover, a novel feature generator, self-calibrated-based 1-D convolutional neural network (SC-1DCNN), is developed to effectively learn shared feature representations from the process data. The adversarial training with gradient penalty is adopted to guide SC-1DCNN to provide domain-invariant features between the two domains. The testing results on four experimental cases under varying working conditions, including two simulation cases and two real cases, have demonstrated the effectiveness of AJDA in process fault diagnosis.

Identifying multimodal imaging‐driven subtypes in mild cognitive impairment using deep multiview learning

AbstractBackgroundMultimodal neuroimaging data can provide complementary information that a single modality cannot about neurodegenerative diseases such as Alzheimer's disease (AD). Deep Generalized Canonical Correlation Analysis (DGCCA) is able to learn a shared feature representation from different views of data by applying non‐linear transformation using neural network. We utilize DGCCA to extract maximally correlated components from multi‐modal neuroimaging data to identify potential imaging‐driven MCI subtypes.MethodWe study 308 Mild Cognitive Impairment (MCI) participants (195 early MCI and 113 late MCI) from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), each with voxel level features from FDG PET, amyloid PET (AV45) and structural MRI processed using voxel‐based morphometry (VBM). Six experimental settings are designed to compare single modality with multiview methods ‐ GCCA and DGCCA, see Figure 1. Agglomerative clustering was used to generated 2 clusters with features from each experiment. To investigate differences between the clusters, Wilcoxon rank‐sum tests are conducted on 11 baseline AD biomarkers including 5 cognitive assessments and 6 brain volume measures, from the ADNI QT‐PAD dataset http://www.pi4cs.org/qt‐pad‐challenge.ResultAmong the two multiview methods, DGCCA is able to explain 68.57% variance with 20 features, while GCCA explains 68.66% variance with 94 features. To evaluate the potential subtypes from clustering, the Calinski‐Harabasz (CH) score, Silhouette score and adjusted mutual information (AMI) score are computed, see Table 1. AV45 generates the best defined clusters, where DGCCA generates clusters with quality comparable to single modality features. In our QT analysis, clusters from FDG and DGCCA features show differential measure in all biomarkers where DGCCA learns from multimodal data, see Figure 2.ConclusionDGCCA is able to learn maximally correlated features from multimodal neuroimaging data with reduced dimensionality, and explain more variance than its linear counterpart GCCA. Cluster analysis shows these imaging‐driven MCI subtypes are different from the currently diagnosis with differential QT measures, by incorporating complementary information from 3 imaging modalities. DGCCA shows to be an effective feature learning method, and this multiview learning framework can identify potentially novel MCI subtypes to facilitate early detection of AD.

Hyperspectral band selection via region-aware latent features fusion based clustering

Band selection is one of the most effective methods to reduce the band redundancy of hyperspectral images (HSIs). Most existing band selection methods tend to regard each band as a whole, and then explore the band redundancy with the pixel-wise features directly. However, since the regions of HSIs corresponding to different objects have diverse spectral properties and spatial structure, such above scheme limits the performance of hyperspectral band selection due to the lack of spatial information. To address above issues, a novel band selection method via region-aware latent features fusion based clustering (RLFFC) is proposed. Specifically, we employ the superpixel segmentation to segment HSIs into multiple regions so that the spatial information of HSIs can be fully preserved. In order to capture the priori information, we construct its corresponding Laplacian matrix from which a group of low dimensional latent features are generated to further enhance the separability among different bands. Then, a shared latent feature representation of HSIs is obtained by fusing region-aware latent features to effectively capture the band redundancy of HSIs. Finally, the k-means clustering algorithm is utilized to obtain the index of the selected bands from the shared latent feature representation. As a result, the spectral and spatial properties are well exploited in the proposed method. Extensive experiments on four public hyperspectral datasets show that the proposed method achieves superior performance when compared with other state-of-the-art ones.

Singular value penalization based adversarial domain adaptation for fault diagnosis of rolling bearings

Machinery fault diagnosis tasks have been well-addressed when sufficient labeled data of the tested machines are available. However, intelligent fault diagnosis methods trained with such labeled data generally fail in classifying unlabeled data due to data distribution discrepancy. To tackle this problem, a novel transfer learning method called singular value penalization adversarial network is proposed for fault diagnosis of rolling bearings, which first constructs a feature extractor to learn the shared feature representations between the source and target domains automatically, and then plays an adversarial training game between feature extractor and domain discriminator to reduce the domain shift and achieve domain adaptation. In particular, a new singular value penalization term is proposed and integrated into the domain adversarial network, which balances the distributions of largest singular values and boosts the discriminability of feature representations. The extensive transfer diagnosis experiments on two rolling bearing datasets validate the effectiveness and robustness of the proposed method, and the results show that our approach is superior to other methods under unlabeled data.

Multisource domain factorization network for cross-domain fault diagnosis of rotating machinery: An unsupervised multisource domain adaptation method

Unsupervised domain adaptation (DA) provides a promising approach for tackling fault diagnosis tasks of target datasets without labeled data and has been actively studied in recent years. Most of them focus only on single-source DA, compared to multisource DA (MDA), which has remarkable advantages in generalized knowledge learning and generalization performance. Nevertheless, there are very few fault diagnosis studies based on MDA, and it remains challenging to reduce multiple domain shifts to improve diagnostic performance and mitigate negative transfer during learning. To this end, a novel unsupervised MDA-based transfer learning approach called multisource domain factorization network (MDFN) is proposed in this paper, where the generalized diagnosis knowledge is learned from multiple sources and then used for diagnosing the target task. The highlights of MDFN are that the shared-space component analysis and transferability-based entropy penalty strategy are employed to significantly mitigate negative transfer from the two levels of feature representation and instance transferability and effectively learn shared feature representation. Therefore, the MDFN can extract shared features that combine domain-invariance and discriminability, thereby performing better. The results of two experimental cases on six datasets, including cross-operating-condition and cross-component diagnosis tasks, validate the effectiveness and superiority of the proposed method.

TUGDA: task uncertainty guided domain adaptation for robust generalization of cancer drug response prediction from in vitro to in vivo settings.

MotivationLarge-scale cancer omics studies have highlighted the diversity of patient molecular profiles and the importance of leveraging this information to deliver the right drug to the right patient at the right time. Key challenges in learning predictive models for this include the high-dimensionality of omics data and heterogeneity in biological and clinical factors affecting patient response. The use of multi-task learning techniques has been widely explored to address dataset limitations for in vitro drug response models, while domain adaptation (DA) has been employed to extend them to predict in vivo response. In both of these transfer learning settings, noisy data for some tasks (or domains) can substantially reduce the performance for others compared to single-task (domain) learners, i.e. lead to negative transfer (NT).ResultsWe describe a novel multi-task unsupervised DA method (TUGDA) that addresses these limitations in a unified framework by quantifying uncertainty in predictors and weighting their influence on shared feature representations. TUGDA’s ability to rely more on predictors with low-uncertainty allowed it to notably reduce cases of NT for in vitro models (94% overall) compared to state-of-the-art methods. For DA to in vivo settings, TUGDA improved over previous methods for patient-derived xenografts (9 out of 14 drugs) as well as patient datasets (significant associations in 9 out of 22 drugs). TUGDA’s ability to avoid NT thus provides a key capability as we try to integrate diverse drug-response datasets to build consistent predictive models with in vivo utility.Availabilityand implementation https://github.com/CSB5/TUGDA.Supplementary information Supplementary data are available at Bioinformatics online.

A novel gradient foster shared-representation convolutional network optimization for multi-modalities

Significant growth has been made with multi-modal data as its entrance in the field of deep learning; whereas, Convolutional Neural Network (CNN) provides sufficient training data to develop a representative encrusted image. Yet, the multi-modality approach in CNN affect the performance by slowly converge the variance along with high-dimensionality, heterogeneity and non-aconvex optimization problems. To abridge these issues, a novel Gradient Foster Shared-representation Convolutional Network (GFSCN) framework is proposed, which improve and optimize the performance interms of accuracy and dimensionality reduction. Initially, the framework incorporates a multiple scant weighted de-noising autoencoder to solve the heterogeneity problem and reduces the dimensionality of data by transforming shared feature representation. Consequently, the work integrated enhanced stochastic variance reduced ascension approach. This approach diminishes the non-convex optimization problem through integrating two gradients consuming mini-batches, which reduced the loss function thereby achieves faster convergence even with the usage of larger dataset. Thus, the proposed framework achieves better performance in terms of achieving utmost accuracy with faster convergence and reduced variance.

PTP: Parallelized Tracking and Prediction With Graph Neural Networks and Diversity Sampling

Multi-object tracking (MOT) and trajectory prediction are two critical components in modern 3D perception systems that require accurate modeling of multi-agent interaction. We hypothesize that it is beneficial to unify both tasks under one framework in order to learn a shared feature representation of agent interaction. Furthermore, instead of performing tracking and prediction sequentially which can propagate errors from tracking to prediction, we propose a parallelized framework to mitigate the issue. Also, our parallel track-forecast framework incorporates two additional novel computational units. First, we use a feature interaction technique by introducing Graph Neural Networks (GNNs) to capture the way in which agents interact with one another. The GNN is able to improve discriminative feature learning for MOT association and provide socially-aware contexts for trajectory prediction. Second, we use a diversity sampling function to improve the quality and diversity of our forecasted trajectories. The learned sampling function is trained to efficiently extract a variety of outcomes from a generative trajectory distribution and helps avoid the problem of generating duplicate trajectory samples. We evaluate on KITTI and nuScenes datasets showing that our method with socially-aware feature learning and diversity sampling achieves new state-of-the-art performance on 3D MOT and trajectory prediction. Project website is: http://www.xinshuoweng.com/projects/PTP.

Body parts relevance learning via expectation–maximization for human pose estimation

Recently, most existing human pose estimation methods fuse multi-stage convolutional modules to learn a shared feature representation. In this paper, we propose a expectation–maximization (EM) mapping-based network to learn specific related body parts for human pose estimation, named EMposeNet. It maps specific feature of related parts from the original fully shared feature space. From the perspective of multi-task learning, we can regard the task of human pose estimation as a homogeneous multi-task learning. Sharing features among related tasks can result in a more compact model and better generalization ability. However, sharing features for those unrelated or weakly related tasks will deteriorate the estimation performance. Our proposed method aims at performing EM algorithm to learn the related body part, where the predicted keypoint heatmap is potentially more accurate and spatially more precise. We conduct extensive experiments on two benchmark datasets, including the MSCOCO keypoint detection dataset and the MPII human pose dataset, to empirically demonstrate the validity of the proposed method. The results on such two benchmark datasets show that the proposed approach achieves a competitive performance.

Multi-task Learning-based All-in-one Collaboration Framework for Degraded Image Super-resolution

In this article, we address the degraded image super-resolution problem in a multi-task learning (MTL) manner. To better share representations between multiple tasks, we propose an all-in-one collaboration framework (ACF) with a learnable “junction” unit to handle two major problems that exist in MTL—“How to share” and “How much to share.” Specifically, ACF consists of a sharing phase and a reconstruction phase. Considering the intrinsic characteristic of multiple image degradations, we propose to first deal with the compression artifact, motion blur, and spatial structure information of the input image in parallel under a three-branch architecture in the sharing phase. Subsequently, in the reconstruction phase, we up-sample the previous features for high-resolution image reconstruction with a channel-wise and spatial attention mechanism. To coordinate two phases, we introduce a learnable “junction” unit with a dual-voting mechanism to selectively filter or preserve shared feature representations that come from sharing phase, learning an optimal combination for the following reconstruction phase. Finally, a curriculum learning-based training scheme is further proposed to improve the convergence of the whole framework. Extensive experimental results on synthetic and real-world low-resolution images show that the proposed all-in-one collaboration framework not only produces favorable high-resolution results while removing serious degradation, but also has high computational efficiency, outperforming state-of-the-art methods. We also have applied ACF to some image-quality sensitive practical task, such as pose estimation, to improve estimation accuracy of low-resolution images.

Knowledge-aware multi-center clinical dataset adaptation: Problem, method, and application

Adaptable utilization of clinical data collected from multiple centers, prompted by the need to overcome the shifts between the dataset distributions, and exploit these different datasets for potential clinical applications, has received significant attention in recent years. In this study, we propose a novel approach to this task by infusing an external knowledge graph (KG) into multi-center clinical data mining. Specifically, we propose an adversarial learning model to capture shared patient feature representations from multi-center heterogeneous clinical datasets, and employ an external KG to enrich the semantics of the patient sample by providing both clinical center-specific and center-general knowledge features, which are trained with a graph convolutional autoencoder. We evaluate the proposed model on a real clinical dataset extracted from the general cardiology wards of a Chinese hospital and a well-known public clinical dataset (MIMIC III, pertaining to ICU clinical settings) for the task of predicting acute kidney injury in patients with heart failure. The achieved experimental results demonstrate the efficacy of our proposed model.

Exploiting interaction of fine and coarse features and attribute co-occurrence for person attribute recognition

Person attribute recognition, i.e., the prediction of a fixed set of semantic attributes given an image of a person, becomes an important topic in the field of computer vision. Recently, methods based on convolutional neural networks have shown outstanding performance in this area. They usually employ a CNN network to mine the shared feature representation followed by several layers for attribute classification. To improve the representation ability of the model, many methods element-add or concatenate coarse and fine feature maps to fuse information at different feature levels. However, these methods didn’t fully exploit the interaction of multi-level convolutional feature maps for person attribute analysis and not consider the correlation of attributes for the same person. In this paper, we introduce a kind of correlation feature, which exploits the high order interaction of coarse and fine feature maps to capture the robust feature representation from multi-level convolution layers as the image representation for person attribute recognition. Moreover, we propose an intraperson attribute loss to explicitly model the correlation of attributes for the same person. We experiment our proposed model on CIFAR-10 dataset, Berkeley Human Attributes dataset, PA-100 K dataset, and experimental results show the better performance of the feature representation and the effectiveness of intra-person attribute loss.

Homogeneous-to-Heterogeneous: Unsupervised Learning for RGB-Infrared Person Re-Identification

RGB-Infrared (RGB-IR) cross-modality person re-identification (re-ID) is attracting more and more attention due to requirements for 24-h scene surveillance. However, the high cost of labeling person identities of an RGB-IR dataset largely limits the scalability of supervised models in real-world scenarios. In this paper, we study the unsupervised RGB-IR person re-ID problem (or briefly uRGB-IR re-ID) in which no identity annotations are available in RGB-IR cross-modality datasets. Considering that intra-modality (i.e., RGB-RGB or IR-IR) re-ID is much easier than cross-modality re-ID and can provide shared knowledge for RGB-IR re-ID, we propose a two-stage method to solve the uRGB-IR re-ID, namely homogeneous-to-heterogeneous learning. In the first stage, the unsupervised self-learning method is conducted to learn the intra-modality feature representation and to generate the pseudo-labeled identities of person images separately for each modality. In the second stage, heterogeneous learning is used to learn a shared discriminative feature representation by distilling the knowledge from intra-modality pseudo-labels, to align two modalities via a modality-based consistent learning module, and finally to target modality-invariant learning via a pseudo-labeled positive instance selection module. With the use of homogeneous-to-heterogeneous learning, the proposed unsupervised framework greatly reduces the modality gap and thus learns a robust feature representation against RGB and infrared modalities, leading to promising accuracy. We also propose a novel cross-modality re-ranking approach that includes a self-modality search and a cycle-modality search to tailor the uRGB-IR re-ID. Unlike conventional re-ranking, the proposed re-ranking method takes a modality-based constraint into re-ranking and thus can select more reliable nearest neighbors, which greatly improves uRGB-IR re-ID. The experimental results demonstrate the superiority of our approach on the SYSU-MM01 and RegDB datasets.

Dual Alignment Self-Supervised Incomplete Multi-View Subspace Clustering Network

Incomplete multi-view clustering has attracted much attention in decade years. To date, most of the remarkable achievements, however, exploit shallow models to learn shared feature representations based on incomplete views. Although some deep learning methods have been proposed to solve this issue, the existing ones still have the following problems: 1) The consistency between views is ignored, which will have serious negative impacts on incomplete multi-view learning. 2) The learned features do not have sufficient cluster-friendliness, that is, the tightness within clusters and the repulsiveness between clusters are not fully considered. To tackle the above shortcomings, we propose a Dual Alignment Self-supervised Incomplete Multi-view Subspace Clustering network (DASIMSC) in this paper. Specifically, the manifold alignment constraint and consistency alignment constraint are integrated with the autoencoder to preserve the compact inherent local structure within the view and the consistency semantics between incomplete views, respectively. Moreover, a self-expression layer coupled with a spectral clustering module is designed to naturally separate different types of data, leveraging the current clustering results to supervise subspace learning, which excludes inter-cluster. Experimental results on several datasets show that our algorithm outperforms all compared state-of-the-arts.

Tasks Integrated Networks: Joint Detection and Retrieval for Image Search

The traditional object (person) retrieval (re-identification) task aims to learn a discriminative feature representation with intra-similarity and inter-dissimilarity, which supposes that the objects in an image are manually or automatically pre-cropped exactly. However, in many real-world searching scenarios (e.g., video surveillance), the objects (e.g., persons, vehicles, etc.) are seldom accurately detected or annotated. Therefore, object-level retrieval becomes intractable without bounding-box annotation, which leads to a new but challenging topic, i.e., image-level search with multi-task integration of joint detection and retrieval. In this paper, to address the image search issue, we first introduce an end-to-end Integrated Net (I-Net), which has three merits: 1) A Siamese architecture and an on-line pairing strategy for similar and dissimilar objects in the given images are designed. Benefited by the Siamese structure, I-Net learns the shared feature representation, because, on which, both object detection and classification tasks are handled. 2) A novel on-line pairing (OLP) loss is introduced with a dynamic feature dictionary, which alleviates the multi-task training stagnation problem, by automatically generating a number of negative pairs to restrict the positives. 3) A hard example priority (HEP) based softmax loss is proposed to improve the robustness of classification task by selecting hard categories. The shared feature representation of I-Net may restrict the task-specific flexibility and learning capability between detection and retrieval tasks. Therefore, with the philosophy of divide and conquer, we further propose an improved I-Net, called DC-I-Net, which makes two new contributions: 1) two modules are tailored to handle different tasks separately in the integrated framework, such that the task specification is guaranteed. 2) A class-center guided HEP loss (C 2HEP) by exploiting the stored class centers is proposed, such that the intra-similarity and inter-dissimilarity can be captured for ultimate retrieval. Extensive experiments on famous image-level search oriented benchmark datasets, such as CUHK-SYSU dataset and PRW dataset for person search and the large-scale WebTattoo dataset for tattoo search, demonstrate that the proposed DC-I-Net outperforms the state-of-the-art tasks-integrated and tasks-separated image search models.

DeepDistance: A multi-task deep regression model for cell detection in inverted microscopy images

This paper presents a new deep regression model, which we call DeepDistance, for cell detection in images acquired with inverted microscopy. This model considers cell detection as a task of finding most probable locations that suggest cell centers in an image. It represents this main task with a regression task of learning an inner distance metric. However, different than the previously reported regression based methods, the DeepDistance model proposes to approach its learning as a multi-task regression problem where multiple tasks are learned by using shared feature representations. To this end, it defines a secondary metric, normalized outer distance, to represent a different aspect of the problem and proposes to define its learning as complementary to the main cell detection task. In order to learn these two complementary tasks more effectively, the DeepDistance model designs a fully convolutional network (FCN) with a shared encoder path and end-to-end trains this FCN to concurrently learn the tasks in parallel. For further performance improvement on the main task, this paper also presents an extended version of the DeepDistance model that includes an auxiliary classification task and learns it in parallel to the two regression tasks by also sharing feature representations with them. DeepDistance uses the inner distances estimated by these FCNs in a detection algorithm to locate individual cells in a given image. In addition to this detection algorithm, this paper also suggests a cell segmentation algorithm that employs the estimated maps to find cell boundaries. Our experiments on three different human cell lines reveal that the proposed multi-task learning models, the DeepDistance model and its extended version, successfully identify the locations of cell as well as delineate their boundaries, even for the cell line that was not used in training, and improve the results of its counterparts.

Modality-specific and shared generative adversarial network for cross-modal retrieval

Cross-modal retrieval aims to realize accurate and flexible retrieval across different modalities of data, e.g., image and text, which has achieved significant progress in recent years, especially since generative adversarial networks (GAN) were used. However, there still exists much room for improvement. How to jointly extract and utilize both the modality-specific (complementarity) and modality-shared (correlation) features effectively has not been well studied. In this paper, we propose an approach named Modality-Specific and Shared Generative Adversarial Network (MS2GAN) for cross-modal retrieval. The network architecture consists of two sub-networks that aim to learn modality-specific features for each modality, followed by a common sub-network that aims to learn the modality-shared features for each modality. Network training is guided by the adversarial scheme between the generative and discriminative models. The generative model learns to predict the semantic labels of features, model the inter- and intra-modal similarity with label information, and ensure the difference between the modality-specific and modality-shared features, while the discriminative model learns to classify the modality of features. The learned modality-specific and shared feature representations are jointly used for retrieval. Experiments on three widely used benchmark multi-modal datasets demonstrate that MS2GAN can outperform state-of-the-art related works.