Weakly-supervised Learning Research Articles

Weakly supervised learning in neural encoding for the position of the moving finger of a macaque

The problem of neural decoding is essential for the realization of a neural interface. In this study, the position of the moving finger of a macaque was directly decoded through the neuron spike signals in the motor cortex, instead of relying on the synergy of the related muscle tissues around the body, also known as neural decoding. Currently, supervised learning is the most commonly employed method for this purpose. However, based on existing technologies, unsupervised learning with regression causes excessive errors. To solve this problem, weakly supervised learning (WSL) was used to correct the predicted position of the moving finger of a macaque in unsupervised training. Then, the corrected finger position was further used to train and accurately fit the weight parameters. We then utilized public data to evaluate the decoding performance of the Kalman filter (KF) and the expectation maximization (EM) algorithms in the WSL model. Unlike in previous methods, in WSL, the only available information is that the finger has moved to four areas in the plane, instead of the actual track value. When compared to the supervised models, the WSL decoding performance only differs by approximately 0.4%. This result improves by 41.3% relative to unsupervised models in the two-dimensional plane. The investigated approach overcomes the instability and inaccuracy of unsupervised learning. What’s more, the method in the paper also verified that the unsupervised encoding and decoding technology of neuronal signals is related to the range of external activities, rather than having a priori specific location.

Deep Reinforcement Learning for Weakly-Supervised Lymph Node Segmentation in CT Images.

Accurate and automated lymph node segmentation is pivotal for quantitatively accessing disease progression and potential therapeutics. The complex variation of lymph node morphology and the difficulty of acquiring voxel-wise manual annotations make lymph node segmentation a challenging task. Since the Response Evaluation Criteria in Solid Tumors (RECIST) annotation, which indicates the location, length, and width of a lymph node, is commonly available in hospital data archives, we advocate to use RECIST annotations as the supervision, and thus formulate this segmentation task into a weakly-supervised learning problem. In this paper, we propose a deep reinforcement learning-based lymph node segmentation (DRL-LNS) model. Based on RECIST annotations, we segment RECIST-slices in an unsupervised way to produce pseudo ground truths, which are then used to train U-Net as a segmentation network. Next, we train a DRL model, in which the segmentation network interacts with the policy network to optimize the lymph node bounding boxes and segmentation results simultaneously. The proposed DRL-LNS model was evaluated against three widely used image segmentation networks on a public thoracoabdominal Computed Tomography (CT) dataset that contains 984 3D lymph nodes, and achieves the mean Dice similarity coefficient (DSC) of 77.17% and the mean Intersection over Union (IoU) of 64.78% in the four-fold cross-validation. Our results suggest that the DRL-based bounding box prediction strategy outperforms the label propagation strategy and the proposed DRL-LNS model is able to achieve the state-of-the-art performance on this weakly-supervised lymph node segmentation task.

Learning multiple instance deep quality representation for robust object tracking

Robustly tracking various objects within a video stream with complex objects and backgrounds is a useful technique in next generation computer vision systems. However, in practice, it is difficult to design a successful video-based object tracking system due to the varied light conditions, possible occlusions, and fast-moving objects. In this work, a novel weakly-supervised and quality-guided visual object tracking model is proposed, wherein the key is a bidirectional long short-term memory recurrent neural network (BLSTM-RNN) that captures the feature sequence and predicts the quality score of each candidate window. More specifically, given a rich set of training videos annotated with the target objects, a weakly-supervised learning algorithm is first used to project all the candidate window features onto the semantic space. Next, we propose a two-stage algorithm to select the key frames from the video sequences, where both the shallow and deep filtering operations are conducted. Subsequently, the so-called BLSTM-RNN is proposed to characterize the feature sequence temporally, based on which the maximally possible object window can be calculated and finally output. In our experiment, a large video dataset containing 2045 NBA regular seasons and playoff basketball games was compiled. Based on this, a comparative study is conducted between the proposed algorithm and state-of-the-art video tracking methods. Extensive visualization results and comparative tracking precisions show the competitiveness of the proposed method.

Weakly-supervised learning for lung carcinoma classification using deep learning

Lung cancer is one of the major causes of cancer-related deaths in many countries around the world, and its histopathological diagnosis is crucial for deciding on optimum treatment strategies. Recently, Artificial Intelligence (AI) deep learning models have been widely shown to be useful in various medical fields, particularly image and pathological diagnoses; however, AI models for the pathological diagnosis of pulmonary lesions that have been validated on large-scale test sets are yet to be seen. We trained a Convolution Neural Network (CNN) based on the EfficientNet-B3 architecture, using transfer learning and weakly-supervised learning, to predict carcinoma in Whole Slide Images (WSIs) using a training dataset of 3,554 WSIs. We obtained highly promising results for differentiating between lung carcinoma and non-neoplastic with high Receiver Operator Curve (ROC) area under the curves (AUCs) on four independent test sets (ROC AUCs of 0.975, 0.974, 0.988, and 0.981, respectively). Development and validation of algorithms such as ours are important initial steps in the development of software suites that could be adopted in routine pathological practices and potentially help reduce the burden on pathologists.

Weakly-Supervised Vessel Detection in Ultra-Widefield Fundus Photography via Iterative Multi-Modal Registration and Learning.

We propose a deep-learning based annotation-efficient framework for vessel detection in ultra-widefield (UWF) fundus photography (FP) that does not require de novo labeled UWF FP vessel maps. Our approach utilizes concurrently captured UWF fluorescein angiography (FA) images, for which effective deep learning approaches have recently become available, and iterates between a multi-modal registration step and a weakly-supervised learning step. In the registration step, the UWF FA vessel maps detected with a pre-trained deep neural network (DNN) are registered with the UWF FP via parametric chamfer alignment. The warped vessel maps can be used as the tentative training data but inevitably contain incorrect (noisy) labels due to the differences between FA and FP modalities and the errors in the registration. In the learning step, a robust learning method is proposed to train DNNs with noisy labels. The detected FP vessel maps are used for the registration in the following iteration. The registration and the vessel detection benefit from each other and are progressively improved. Once trained, the UWF FP vessel detection DNN from the proposed approach allows FP vessel detection without requiring concurrently captured UWF FA images. We validate the proposed framework on a new UWF FP dataset, PRIME-FP20, and on existing narrow-field FP datasets. Experimental evaluation, using both pixel-wise metrics and the CAL metrics designed to provide better agreement with human assessment, shows that the proposed approach provides accurate vessel detection, without requiring manually labeled UWF FP training data.

Robust Semi-Supervised Traffic Sign Recognition via Self-Training and Weakly-Supervised Learning.

Traffic sign recognition is a classification problem that poses challenges for computer vision and machine learning algorithms. Although both computer vision and machine learning techniques have constantly been improved to solve this problem, the sudden rise in the number of unlabeled traffic signs has become even more challenging. Large data collation and labeling are tedious and expensive tasks that demand much time, expert knowledge, and fiscal resources to satisfy the hunger of deep neural networks. Aside from that, the problem of having unbalanced data also poses a greater challenge to computer vision and machine learning algorithms to achieve better performance. These problems raise the need to develop algorithms that can fully exploit a large amount of unlabeled data, use a small amount of labeled samples, and be robust to data imbalance to build an efficient and high-quality classifier. In this work, we propose a novel semi-supervised classification technique that is robust to small and unbalanced data. The framework integrates weakly-supervised learning and self-training with self-paced learning to generate attention maps to augment the training set and utilizes a novel pseudo-label generation and selection algorithm to generate and select pseudo-labeled samples. The method improves the performance by: (1) normalizing the class-wise confidence levels to prevent the model from ignoring hard-to-learn samples, thereby solving the imbalanced data problem; (2) jointly learning a model and optimizing pseudo-labels generated on unlabeled data; and (3) enlarging the training set to satisfy the hunger of deep learning models. Extensive evaluations on two public traffic sign recognition datasets demonstrate the effectiveness of the proposed technique and provide a potential solution for practical applications.

Inaccurate Labels in Weakly-Supervised Deep Learning: Automatic Identification and Correction and Their Impact on Classification Performance.

In data-driven deep learning-based modeling, data quality may substantially influence classification performance. Correct data labeling for deep learning modeling is critical. In weakly-supervised learning, a challenge lies in dealing with potentially inaccurate or mislabeled training data. In this paper, we proposed an automated methodological framework to identify mislabeled data using two metric functions, namely, Cross-entropy Loss that indicates divergence between a prediction and ground truth, and Influence function that reflects the dependence of a model on data. After correcting the identified mislabels, we measured their impact on the classification performance. We also compared the mislabeling effects in three experiments on two different real-world clinical questions. A total of 10,500 images were studied in the contexts of clinical breast density category classification and breast cancer malignancy diagnosis. We used intentionally flipped labels as mislabels to evaluate the proposed method at a varying proportion of mislabeled data included in model training. We also compared the effects of our method to two published schemes for breast density category classification. Experiment results show that when the dataset contains 10% of mislabeled data, our method can automatically identify up to 98% of these mislabeled data by examining/checking the top 30% of the full dataset. Furthermore, we show that correcting the identified mislabels leads to an improvement in the classification performance. Our method provides a feasible solution for weakly-supervised deep learning modeling in dealing with inaccurate labels.

Multiple instance deep learning for weakly-supervised visual object tracking

Intelligently tracking objects with varied shapes, color, lighting conditions, and backgrounds is an extremely useful application in many HCI applications, such as human body motion capture, hand gesture recognition, and virtual reality (VR) games. However, accurately tracking different objects under uncontrolled environments is a tough challenge due to the possibly dynamic object parts, varied lighting conditions, and sophisticated backgrounds. In this work, we propose a novel semantically-aware object tracking framework, wherein the key is weakly-supervised learning paradigm that optimally transfers the video-level semantic tags into various regions. More specifically, give a set of training video clips, each of which is associated with multiple video-level semantic tags, we first propose a weakly-supervised learning algorithm to transfer the semantic tags into various video regions. The key is a MIL (Zhong et al., 2020) [1]-based manifold embedding algorithm that maps the entire video regions into a semantic space, wherein the video-level semantic tags are well encoded. Afterward, for each video region, we use the semantic feature combined with the appearance feature as its representation. We designed a multi-view learning algorithm to optimally fuse the above two types of features. Based on the fused feature, we learn a probabilistic Gaussian mixture model to predict the target probability of each candidate window, where the window with the maximal probability is output as the tracking result. Comprehensive comparative results on a challenging pedestrian tracking task as well as the human hand gesture recognition have demonstrated the effectiveness of our method. Moreover, visualized tracking results have shown that non-rigid objects with moderate occlusions can be well localized by our method.

On the Importance of Visual Context for Data Augmentation in Scene Understanding.

Performing data augmentation for learning deep neural networks is known to be important for training visual recognition systems. By artificially increasing the number of training examples, it helps reducing overfitting and improves generalization. While simple image transformations can already improve predictive performance in most vision tasks, larger gains can be obtained by leveraging task-specific prior knowledge. In this work, we consider object detection, semantic and instance segmentation and augment the training images by blending objects in existing scenes, using instance segmentation annotations. We observe that randomly pasting objects on images hurts the performance, unless the object is placed in the right context. To resolve this issue, we propose an explicit context model by using a convolutional neural network, which predicts whether an image region is suitable for placing a given object or not. In our experiments, we show that our approach is able to improve object detection, semantic and instance segmentation on the PASCAL VOC12 and COCO datasets, with significant gains in a limited annotation scenario, i.e., when only one category is annotated. We also show that the method is not limited to datasets that come with expensive pixel-wise instance annotations and can be used when only bounding boxes are available, by employing weakly-supervised learning for instance masks approximation.

A Survey on Temporal Action Localization

Temporal action localization is one of the most crucial and challenging problems for video understanding in computer vision. It has received a lot of attention in recent years because of the extensive application of daily life. Temporal action localization has made some significant progress, especially with the development of deep learning recently. And more demand is for temporal action localization in untrimmed videos. In this paper, our target is to survey the state-of-the-art techniques and models for video temporal action localization. It mainly includes the related techniques, some benchmark datasets and the evaluation metrics of temporal action localization. In addition, we summarize temporal action localization from two aspects: fully-supervised learning and weakly-supervised learning. And we list several representative works and compare their performances respectively. Finally, we make some deep analysis and propose potential research directions, and conclude the survey.

Learning to Recognize Thoracic Disease in Chest X-Rays With Knowledge-Guided Deep Zoom Neural Networks

Automatic and accurate thorax disease diagnosis in Chest X-ray (CXR) image plays an essential role in clinical assist analysis. However, due to its imaging noise regions and the similarity of visual features between diseases and their surroundings, the precise analysis of thoracic disease becomes a challenging problem. In this study, we propose a novel knowledge-guided deep zoom neural network (KGZNet) which is a data-driven model. Our approach leverage prior medical knowledge to guide its training process, due to thoracic diseases typically limit within the lung regions. Also, we utilized weakly-supervised learning (WSL) to search for finer regions without using annotated samples. Learning on each scale consists of a classification sub-network. The KGZNet starts from global images, and iteratively generates discriminative part from coarse to fine; while a finer scale sub-network takes as input an amplified attended discriminative region from previous scales in a recurrent way. Specifically, we first train a robust modified U-Net model of lung segmentation and capture the lung area from the original CXR image through the Lung Region Generator. Then, guided by the attention heatmap, we obtain a finer discriminative lesion region from the lung region images by the Lesion Region Generator. Lastly, the most discriminative features knowledge is fused, and the complementary features information is learned for final disease prediction. Extensive experiments demonstrate that our method can effectively leverage discriminative region information, and significantly outperforms the other state-of-the-art methods in the thoracic disease recognition task. Furthermore, the proposed KGZNet can gradually learn the discriminative region from coarse to fine in a mutually reinforced way. The code is will available at: https://github.com/ISSE-AILab/KGZNet.

Hand gesture understanding by weakly-supervised fusing shallow/deep image attributes

Accurately recognizing human hand gestures is a useful component in many modern intelligent systems, such as identification authentication, human–computer interaction, and sign language recognition. Conventional approaches are typically based on shallow visual features and relatively simple backgrounds, which cannot readily recognize partially occluded hand gestures with sophisticated backgrounds. In this work, we propose a unified hand gesture recognition framework by optimally fusing a set of shallow/deep finger-level image attributes, based on which a weakly-supervised ranking algorithm is designed to select semantically salient regions for gesture understanding. More specifically, given a rich number of hand gesture images, we employ the well-known BING object proposal generator to extract hundreds of object patches that potentially draw human visual attention. Since the hundreds of object patches are still too many for building an effective recognition system, a weakly-supervised metric is proposed to rank them by extracting multiple shallow/deep features. And visual semantics are encoded at region-level by transferring the image-level semantic tags into various human gesture image regions by a weakly-supervised learning paradigm Apparently, the top-ranking highly salient object patches are highly indicative to human visual perception of human hand gesture, Thus we extract their ImageNet-CNN features and further concatenate them. Finally, the concatenated deep feature is fed into a multi-class SVM for classifying each hand gesture image into a particular type. Comprehensive experimental validations have demonstrated the effectiveness and robustness of our proposed hybrid-feature-based hand gesture categorization.

Weakly-supervised large-scale image modeling for sport scenes and its applications

Image modeling towards sport scenes plays an important role in sport image classification and analysis. Traditional algorithms for sport image modeling required carefully hand-crafted features, which cannot be popularized in practical application, especially with the emergence of massive-scale data. Weakly-supervised learning algorithms have shown effectiveness in modeling data with image-level labels. Thus, in this paper, we propose a weakly-supervised learning based method for sport image modeling without utilizing bounding box annotations, which can be used for various sport image applications. More specifically, we first collect large-scale sport images from existing datasets and Internet, and we annotate them at image-level labels. Subsequently, we leverage region proposal generation algorithm to select discriminative regions that can effectively represent the category of images. Each region is fed into a pre-trained CNN architecture to extract deep representation. Afterwards, we design an improved multiple discriminant analysis (MDA) algorithm to project these datapoints to a subspace that can more easily to distinguish different sport categories. Comprehensive experiments have shown the effectiveness and robustness of our proposed method.

Weakly-Supervised Learning of Category-Specific 3D Object Shapes.

Category-specific 3D object shape models have greatly boosted the recent advances in object detection, recognition and segmentation. However, even the most advanced approach for learning 3D object shapes still requires heavy manual annotations on large-scale 2D images. Such annotations include object categories, object keypoints, and figure-ground segmentation for the instances in each image. In particular, annotating figure-ground segmentation is unbearably labor-intensive and time-consuming. To address this problem, this paper devotes to learn category-specific 3D shape models under weak supervision, where only object categories and keypoints are required to be manually annotated on the training 2D images. By exploring the underlying relationship between two tasks: object segmentation and category-specific 3D shape reconstruction, we propose a novel weakly-supervised learning framework to jointly address these two tasks and combine them to boost the final performance of the learned 3D shape models. Moreover, learning without using figure-ground segmentation leads to ambiguous solutions. To this end, we develop the confidence weighting schemes in the viewpoint estimation and 3D shape learning procedure. These schemes effectively reduce the confusion caused by the noisy data and thus increase the chances for recovering more reliable 3D object shapes. Comprehensive experiments on the challenging PASCAL VOC benchmark show that our framework achieves comparable performance with the state-of-the-art methods that use expensive manual segmentation-level annotations. In addition, our experiments also demonstrate that our 3D shape models improve object segmentation performance.

Target Discrimination Based on Weakly Supervised Learning for High-Resolution SAR Images in Complex Scenes

To design a highly automatic and practical discrimination method for high-resolution synthetic aperture radar (SAR) images in complex scenes, a novel target discrimination framework based on weakly supervised learning (WSL) of the mid-level features is proposed in this article. First, we extract the dense SAR scale-invariant feature transform (SAR-SIFT) features of the candidate regions obtained from the detected SAR images. Then, the dense SAR-SIFT descriptors are transformed into richer mid-level features by coding and pooling. Finally, the mid-level features are input into a WSL-based target discrimination method, where the training set is initially selected by the unsupervised latent Dirichlet allocation (LDA) and iteratively updated by the linear support vector machine (SVM) discriminator. In the proposed method, only the image-level annotations (weak labels), which indicate whether the images containing the targets of interest or not, are required. By introducing WSL, the manual annotations of target regions from SAR images can be avoided, which is generally expensive in complex scenes and may tend to be less accurate and unreliable for the occluded or camouflaged targets. The comprehensive and specific experiments on the measured SAR data have demonstrated the effectiveness of the proposed method in benchmarking with the supervised learning-based linear SVM and linear support vector data description (SVDD) discriminators.

A component-based video content representation for action recognition

This paper investigates the challenging problem of action recognition in videos and proposes a new component-based approach for video content representation. Although satisfactory performance for action recognition has already been obtained for certain scenarios, many of the existing solutions require fully-annotated video datasets in which region of the activity in each frame is specified by a bounding box. Another group of methods require auxiliary techniques to extract human-related areas in the video frames before being able to accurately recognize actions. In this paper, a Weakly-Supervised Learning (WSL) framework is introduced that eliminates the need for per-frame annotations and learns video representations that improve recognition accuracy and also highlights the activity related regions within each frame. To this end, two new representation ideas are proposed, one focus on representing the main components of an action, i.e. actionness regions, and the other focus on encoding the background context to represent general and holistic cues. A three-stream CNN is developed, which takes the two proposed representations and combines them with a motion-encoding stream. Temporal cues in each of the three different streams are modeled through LSTM, and finally fully-connected neural network layers are used to fuse various streams and produce the final video representation. Experimental results on four challenging datasets, demonstrate that the proposed Component-based Multi-stream CNN model (CM-CNN), trained on a WSL setting, outperforms the state-of-the-art in action recognition, even the fully-supervised approaches.

Weakly-supervised learning approach for potato defects segmentation

Rigorous quality analysis of potatoes is essential to define their market price. Manual approaches to detect skin defects of this tuber are laborious, subjective and time-consuming. In this paper, we introduce a weakly-supervised learning method to classify, localize and segment potato defects to automate the quality control task. A large and diversified image-level labeled dataset is created including potatoes from six different classes: healthy, damaged, greening, black dot, common scab and black scurf. A convolutional neural network (CNN) is trained to achieve the classification task. Then, we leverage the discriminative regions that appear in the activation maps of the trained CNN to localize the classified defect. A coarse-to-fine segmentation method is proposed to obtain a more precise defect size. Based on this segmentation, a classification according to the severity of the defect is done, showing the importance of the segmentation phase. Experimental results demonstrate that CNN outperforms conventional classifiers. At a final stage, a multi-label multi-class dataset is used to evaluate the whole system, achieving an average precision of 0.91 and an average recall of 0.90.

Vision-Based Freezing of Gait Detection With Anatomic Directed Graph Representation.

Parkinson's disease significantly impacts the life quality of millions of people around the world. While freezing of gait (FoG) is one of the most common symptoms of the disease, it is time consuming and subjective to assess FoG for well-trained experts. Therefore, it is highly desirable to devise computer-aided FoG detection methods for the purpose of objective and time-efficient assessment. In this paper, in line with the gold standard of FoG clinical assessment, which requires video or direct observation, we propose one of the first vision-based methods for automatic FoG detection. To better characterize FoG patterns, instead of learning an overall representation of a video, we propose a novel architecture of graph convolution neural network and represent each video as a directed graph where FoG related candidate regions are the vertices. A weakly-supervised learning strategy and a weighted adjacency matrix estimation layer are proposed to eliminate the resource expensive data annotation required for fully supervised learning. As a result, the interference of visual information irrelevant to FoG, such as gait motion of supporting staff involved in clinical assessments, has been reduced to improve FoG detection performance by identifying the vertices contributing to FoG events. To further improve the performance, the global context of a clinical video is also considered and several fusion strategies with graph predictions are investigated. Experimental results on more than 100 videos collected from 45 patients during a clinical assessment demonstrated promising performance of our proposed method with an AUC of 0.887.

A Novel Weakly-Supervised Approach for RGB-D-Based Nuclear Waste Object Detection

This paper addresses the problem of RGBD object recognition in real-world applications, where large amounts of annotated training data are typically unavailable. To overcome this problem, we propose a novel, weakly-supervised learning architecture (DCNN-GPC) which combines parametric models (a pair of Deep Convolutional Neural Networks (DCNN) for RGB and D modalities) with non-parametric models (Gaussian Process Classification). Our system is initially trained using a small amount of labeled data, and then automatically prop- agates labels to large-scale unlabeled data. We first run 3D- based objectness detection on RGBD videos to acquire many unlabeled object proposals, and then employ DCNN-GPC to label them. As a result, our multi-modal DCNN can be trained end-to-end using only a small amount of human annotation. Finally, our 3D-based objectness detection and multi-modal DCNN are integrated into a real-time detection and recognition pipeline. In our approach, bounding-box annotations are not required and boundary-aware detection is achieved. We also propose a novel way to pretrain a DCNN for the depth modality, by training on virtual depth images projected from CAD models. We pretrain our multi-modal DCNN on public 3D datasets, achieving performance comparable to state-of-the-art methods on Washington RGBS Dataset. We then finetune the network by further training on a small amount of annotated data from our novel dataset of industrial objects (nuclear waste simulants). Our weakly supervised approach has demonstrated to be highly effective in solving a novel RGBD object recognition application which lacks of human annotations.

Varifocal-Net: A Chromosome Classification Approach Using Deep Convolutional Networks.

Chromosome classification is critical for karyotyping in abnormality diagnosis. To expedite the diagnosis, we present a novel method named Varifocal-Net for simultaneous classification of chromosome's type and polarity using deep convolutional networks. The approach consists of one global-scale network (G-Net) and one local-scale network (L-Net). It follows three stages. The first stage is to learn both global and local features. We extract global features and detect finer local regions via the G-Net. By proposing a varifocal mechanism, we zoom into local parts and extract local features via the L-Net. Residual learning and multi-task learning strategies are utilized to promote high-level feature extraction. The detection of discriminative local parts is fulfilled by a localization subnet of the G-Net, whose training process involves both supervised and weakly supervised learning. The second stage is to build two multi-layer perceptron classifiers that exploit features of both two scales to boost classification performance. The third stage is to introduce a dispatch strategy of assigning each chromosome to a type within each patient case, by utilizing the domain knowledge of karyotyping. The evaluation results from 1909 karyotyping cases showed that the proposed Varifocal-Net achieved the highest accuracy per patient case (%) of 99.2 for both type and polarity tasks. It outperformed state-of-the-art methods, demonstrating the effectiveness of our varifocal mechanism, multi-scale feature ensemble, and dispatch strategy. The proposed method has been applied to assist practical karyotype diagnosis.