Explicit Supervision Research Articles

Spatial-temporal attention for video-based assessment of intraoperative surgical skill

Accurate, unbiased, and reproducible assessment of skill is a vital resource for surgeons throughout their career. The objective in this research is to develop and validate algorithms for video-based assessment of intraoperative surgical skill. Algorithms to classify surgical video into expert or novice categories provide a summative assessment of skill, which is useful for evaluating surgeons at discrete time points in their training or certification of surgeons. Using a spatial-temporal neural network architecture, we tested the hypothesis that explicit supervision of spatial attention supervised by instrument tip locations improves the algorithm’s generalizability to unseen dataset. The best performing model had an area under the receiver operating characteristic curve (AUC) of 0.88. Augmenting the network with supervision of spatial attention improved specificity of its predictions (with small changes in sensitivity and AUC) and led to improved measures of discrimination when tested with unseen dataset. Our findings show that explicit supervision of attention learned from images using instrument tip locations can improve performance of algorithms for objective video-based assessment of surgical skill.

Boundary-aware convolutional attention network for liver segmentation in ultrasound images

Liver ultrasound is widely used in clinical practice due to its advantages of non-invasiveness, non-radiation, and real-time imaging. Accurate segmentation of the liver region in ultrasound images is essential for accelerating the auxiliary diagnosis of liver-related diseases. This paper proposes BACANet, a deep learning algorithm designed for real-time liver ultrasound segmentation. Our approach utilizes a lightweight network backbone for liver feature extraction and incorporates a convolutional attention mechanism to enhance the network's ability to capture global contextual information. To improve early localization of liver boundaries, we developed a selective large kernel convolution module for boundary feature extraction and introduced explicit liver boundary supervision. Additionally, we designed an enhanced attention gate to efficiently convey liver body and boundary features to the decoder to enhance the feature representation capability. Experimental results across multiple datasets demonstrate that BACANet effectively completes the task of liver ultrasound segmentation, achieving a balance between inference speed and segmentation accuracy. On a public dataset, BACANet achieved a DSC of 0.921 and an IOU of 0.854. On a private test dataset, BACANet achieved a DSC of 0.950 and an IOU of 0.907, with an inference time of approximately 0.32 s per image on a CPU processor.

Modality adaptation via feature difference learning for depth human parsing

In the field of human parsing, depth data offers unique advantages over RGB data due to its illumination invariance and geometric detail, which motivates us to explore human parsing with only depth input. However, depth data is challenging to collect at scale due to the specialized equipment required. In contrast, RGB data is readily available in large quantities, presenting an opportunity to enhance depth-only parsing models with semantic knowledge learned from RGB data. However, fully finetuning the RGB-pretrained encoder leads to high training costs and inflexible domain generalization, while keeping the encoder frozen suffers from a large RGB-depth modality gap and restricts the parsing performance. To alleviate the limitations of these naive approaches, we introduce a Modality Adaptation pipeline via Feature Difference Learning (MAFDL) which leverages the RGB knowledge to facilitate depth human parsing. A Difference-Guided Depth Adapter (DGDA) is proposed within MAFDL to learn the feature differences between RGB and depth modalities, adapting depth features into RGB feature space to bridge the modality gap. Furthermore, we also design a Feature Alignment Constraint (FAC) to impose explicit alignment supervision at pixel and batch levels, making the modality adaptation more comprehensive. Extensive experiments on the NTURGBD-Parsing-4K dataset show that our method surpasses previous state-of-the-art approaches.

Identity-Preserving Face Swapping via Dual Surrogate Generative Models

In this study, we revisit the fundamental setting of face-swapping models and reveal that only using implicit supervision for training leads to the difficulty of advanced methods to preserve the source identity. We propose a novel reverse pseudo-input generation approach to offer supplemental data for training face-swapping models, which addresses the aforementioned issue. Unlike the traditional pseudo-label-based training strategy, we assume that arbitrary real facial images could serve as the ground-truth outputs for the face-swapping network and try to generate corresponding input < source, target > pair data. Specifically, we involve a source-creating surrogate that alters the attributes of the real image while keeping the identity, and a target-creating surrogate intends to synthesize attribute-preserved target images with different identities. Our framework, which utilizes proxy-paired data as explicit supervision to direct the face-swapping training process, partially fulfills a credible and effective optimization direction to boost the identity-preserving capability. We design explicit and implicit adaption strategies to better approximate the explicit supervision for face swapping. Quantitative and qualitative experiments on FF++, FFHQ, and wild images show that our framework could improve the performance of various face-swapping pipelines in terms of visual fidelity and ID preserving. Furthermore, we display applications with our method on re-aging, swappable attribute customization, cross-domain, and video face swapping. Code is available under https://github.com/ICTMCG/CSCS.

Multi-modal trajectory forecasting with Multi-scale Interactions and Multi-pseudo-target Supervision

Trajectory forecasting is crucial for the advancement of autonomous vehicles. While much progress has been made, extant approaches often fall short in accounting for intricate social interactions and the unpredictability of human behavior. This paper introduces a novel multi-modal trajectory forecasting model named Multi-scale Interaction and Multi-pseudo-target Supervision (MIMS). Central to our approach is a multiscale hypergraph that discerns latent interactions across all scales, evaluating both their strength and type. Further enhancing our model’s capabilities is a causal-effect-estimation-based pseudo-target generation method. This facilitates multi-modality modeling in motion forecasting by offering explicit supervision with multiple latent targets. We validate MIMS using the Argoverse Motion Forecasting dataset. The results reveal that MIMS outperforms current state-of-the-art models across various traffic conditions. Specifically, our multiscale hypergraph exhibits a superior capacity for capturing complex spatiotemporal dependencies compared to pairwise methods. Moreover, our multi-pseudo-target supervision eliminates the pattern convergence of multi-modal forecasting.

Neural Networks as Cognitive Models of the Processing of Syntactic Constraints.

Languages are governed by syntactic constraints-structural rules that determine which sentences are grammatical in the language. In English, one such constraint is subject-verb agreement, which dictates that the number of a verb must match the number of its corresponding subject: "the dogs run", but "the dog runs". While this constraint appears to be simple, in practice speakers make agreement errors, particularly when a noun phrase near the verb differs in number from the subject (for example, a speaker might produce the ungrammatical sentence "the key to the cabinets are rusty"). This phenomenon, referred to as agreement attraction, is sensitive to a wide range of properties of the sentence; no single existing model is able to generate predictions for the wide variety of materials studied in the human experimental literature. We explore the viability of neural network language models-broad-coverage systems trained to predict the next word in a corpus-as a framework for addressing this limitation. We analyze the agreement errors made by Long Short-Term Memory (LSTM) networks and compare them to those of humans. The models successfully simulate certain results, such as the so-called number asymmetry and the difference between attraction strength in grammatical and ungrammatical sentences, but failed to simulate others, such as the effect of syntactic distance or notional (conceptual) number. We further evaluate networks trained with explicit syntactic supervision, and find that this form of supervision does not always lead to more human-like syntactic behavior. Finally, we show that the corpus used to train a network significantly affects the pattern of agreement errors produced by the network, and discuss the strengths and limitations of neural networks as a tool for understanding human syntactic processing.

A domain knowledge-informed design space exploration methodology for mechanical layout design

Layout designs have a large number of design variables and various physical constraints. Conventional design exploration approaches are time-consuming and may require human intervention. A unique feature about physical layout designs is the availability of domain knowledge, which can be utilised to speed up the design process. In this paper, we propose a generative deep learning-based design space exploration (DSE) methodology that is capable of learning design constraints in the layout design without explicit supervision. Moreover, it can incorporate domain knowledge in the generated layouts, thereby speeding up the design process. This is realised by constructing a layout generation variational autoencoder (LGVAE) model, which uses a latent space as an interface to generate the layouts. By training the LGVAE model, significantly lower-dimensional representations can be learned compared to the original dimensionality of the design space. Therefore, the number of design variables is greatly reduced. We showcase the performance of the proposed DSE approach by solving the heat source layout design problem encountered in thermal management of chips. Experiments demonstrate that the LGVAE model is capable of generating compressed latent representations that capture the characteristics of the input samples, which makes the DSE cost-effective.

A Critical Analysis of Deep Semi-Supervised Learning Approaches for Enhanced Medical Image Classification

Deep semi-supervised learning (DSSL) is a machine learning paradigm that blends supervised and unsupervised learning techniques to improve the performance of various models in computer vision tasks. Medical image classification plays a crucial role in disease diagnosis, treatment planning, and patient care. However, obtaining labeled medical image data is often expensive and time-consuming for medical practitioners, leading to limited labeled datasets. DSSL techniques aim to address this challenge, particularly in various medical image tasks, to improve model generalization and performance. DSSL models leverage both the labeled information, which provides explicit supervision, and the unlabeled data, which can provide additional information about the underlying data distribution. That offers a practical solution to resource-intensive demands of data annotation, and enhances the model’s ability to generalize across diverse and previously unseen data landscapes. The present study provides a critical review of various DSSL approaches and their effectiveness and challenges in enhancing medical image classification tasks. The study categorized DSSL techniques into six classes: consistency regularization method, deep adversarial method, pseudo-learning method, graph-based method, multi-label method, and hybrid method. Further, a comparative analysis of performance for six considered methods is conducted using existing studies. The referenced studies have employed metrics such as accuracy, sensitivity, specificity, AUC-ROC, and F1 score to evaluate the performance of DSSL methods on different medical image datasets. Additionally, challenges of the datasets, such as heterogeneity, limited labeled data, and model interpretability, were discussed and highlighted in the context of DSSL for medical image classification. The current review provides future directions and considerations to researchers to further address the challenges and take full advantage of these methods in clinical practices.

PURF: Improving teacher representations by imposing smoothness constraints for knowledge distillation

Knowledge distillation is one of the most persuasive approaches to model compression that transfers the representational expertise from large deep-learning teacher models to a small student network. Although numerous techniques have been proposed to improve teacher representations at the logits level, no study has examined the weaknesses in the representations of the teacher at the feature level during distillation. On the other hand, in a trained deep-learning model, all the kernels are not uniformly activated to make specific predictions. Transferring this knowledge may result in a student learning a suboptimal intrinsic distribution and restrict the existing distillation methods from exploiting their highest potential. Motivated by these issues, this study analyses the generalization capability of teachers with or without a uniformly activated channel distribution. Preliminary investigations and theoretical analyses show that partly uniforming or smoothing feature maps offer improved representation that enriches the generalization capability. Based on these observations, it is hypothesized that distillation-based explicit supervision using smoothed feature maps and cross-entropy loss plays a significant role in improving generalization. Hence, this paper proposes a novel technique called Partly Unified Recalibrated Feature (PURF) map distillation. The proposed method recalibrates the feature maps by intercommunicating the representational cues among nearest-neighbor channels. PURF increases the performance of state-of-the-art knowledge distillation (KD) methods across architectures by improving generalization, model compression, few-shot training, transferability, and robustness transfer on standard benchmark datasets. PURF achieves 1.51% average accuracy improvements on seven diverse architectures in image classification tasks. PURF increases the performance of the state-of-the-art knowledge distillation methods by an average accuracy of 1.91% across architectures. Moreover, PURF achieves an average of 2.02%, and 0.96% higher accuracy in transferability and robustness tasks, respectively, on standard benchmark datasets.

AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose Optimization

Neural Radiance Fields (NeRF) have shown promise in generating realistic novel views from sparse scene images. However, existing NeRF approaches often encounter challenges due to the lack of explicit 3D supervision and imprecise camera poses, resulting in suboptimal outcomes. To tackle these issues, we propose AltNeRF---a novel framework designed to create resilient NeRF representations using self-supervised monocular depth estimation (SMDE) from monocular videos, without relying on known camera poses. SMDE in AltNeRF masterfully learns depth and pose priors to regulate NeRF training. The depth prior enriches NeRF's capacity for precise scene geometry depiction, while the pose prior provides a robust starting point for subsequent pose refinement. Moreover, we introduce an alternating algorithm that harmoniously melds NeRF outputs into SMDE through a consistence-driven mechanism, thus enhancing the integrity of depth priors. This alternation empowers AltNeRF to progressively refine NeRF representations, yielding the synthesis of realistic novel views. Extensive experiments showcase the compelling capabilities of AltNeRF in generating high-fidelity and robust novel views that closely resemble reality.

Semantics-Guided Contrastive Network for Zero-Shot Object Detection.

Zero-shot object detection (ZSD), the task that extends conventional detection models to detecting objects from unseen categories, has emerged as a new challenge in computer vision. Most existing approaches tackle the ZSD task with a strict mapping-transfer strategy that may lead to suboptimal ZSD results: 1) the learning process of these models neglects the available semantic information on unseen classes, which can easily bias towards the seen categories; 2) the original visual feature space is not well-structured for the ZSD task due to the lack of discriminative information. To address these issues, we develop a novel Semantics-Guided Contrastive Network for ZSD, named ContrastZSD, a detection framework that first brings contrastive learning mechanism into the realm of zero-shot detection. Particularly, ContrastZSD incorporates two semantics-guided contrastive learning subnets that contrast between region-category and region-region pairs respectively. The pairwise contrastive tasks take advantage of supervision signals derived from both the ground truth label and class similarity information. By performing supervised contrastive learning over those explicit semantic supervision, the model can learn more knowledge about unseen categories to avoid the bias problem to seen concepts, while optimizing the visual data structure to be more discriminative for better visual-semantic alignment. Extensive experiments are conducted on two popular benchmarks for ZSD, i.e., PASCAL VOC and MS COCO. Results show that our method outperforms the previous state-of-the-art on both ZSD and generalized ZSD tasks.

Target Detection Adapting to Spectral Variability in Multi-Temporal Hyperspectral Images Using Implicit Contrastive Learning

Hyperspectral target detection (HTD) is a crucial aspect of remote sensing applications, aiming to identify targets in hyperspectral images (HSIs) based on their known prior spectral signatures. However, the spectral variability resulting from various imaging conditions in multi-temporal hyperspectral images poses a challenge to both classical and deep learning (DL) methods. To overcome the limitations imposed by spectral variability, an implicit contrastive learning-based target detector (ICLTD) is proposed to exploit in-scene spectra in an unsupervised way. First, only prior spectra are utilized for explicit supervision, while an implicit contrastive learning module (ICLM) is designed to normalize the feature distributions of prior and in-scene spectra. This paper theoretically demonstrates that the ICLM can transfer the gradients from prior spectral features to those of in-scene spectra based on their feature similarities and differences. Because of transferred gradient signals, the ICLTD is regularized to extract similar representations for the prior and in-scene target spectra, while augmenting feature differences between the target and background spectra. Additionally, a local spectral similarity constraint (LSSC) is proposed to enhance the capability of scene adaptation by leveraging the spectral similarities among in-scene targets. To validate the performance of the ICLTD under spectral variability, multi-temporal HSIs captured under various imaging conditions are collected to generate prior spectra and in-scene spectra. Comparative evaluations against several DL detectors and classical methods reveal the superior performance of the ICLTD in achieving a balance between target detectability and background suppressibility under spectral variability.

Transformer-Based Spatio-Temporal Analysis for Classification of Aortic Stenosis Severity From Echocardiography Cine Series.

Aortic stenosis (AS) is characterized by restricted motion and calcification of the aortic valve and is the deadliest valvular cardiac disease. Assessment of AS severity is typically done by expert cardiologists using Doppler measurements of valvular flow from echocardiography. However, this limits the assessment of AS to hospitals staffed with experts to provide comprehensive echocardiography service. As accurate Doppler acquisition requires significant clinical training, in this paper, we present a deep learning framework to determine the feasibility of AS detection and severity classification based only on two-dimensional echocardiographic data. We demonstrate that our proposed spatio-temporal architecture effectively and efficiently combines both anatomical features and motion of the aortic valve for AS severity classification. Our model can process cardiac echo cine series of varying length and can identify, without explicit supervision, the frames that are most informative towards the AS diagnosis. We present an empirical study on how the model learns phases of the heart cycle without any supervision and frame-level annotations. Our architecture outperforms state-of-the-art results on a private and a public dataset, achieving 95.2% and 91.5% in AS detection, and 78.1% and 83.8% in AS severity classification on the private and public datasets, respectively. Notably, due to the lack of a large public video dataset for AS, we made slight adjustments to our architecture for the public dataset. Furthermore, our method addresses common problems in training deep networks with clinical ultrasound data, such as a low signal-to-noise ratio and frequently uninformative frames. Our source code is available at: https://github.com/neda77aa/FTC.git.

Region-Based Unsupervised Low-Light Image Enhancement in the Wild With Explicit Domain Supervision

Region-Based Unsupervised Low-Light Image Enhancement in the Wild With Explicit Domain Supervision

Saliency as Pseudo-Pixel Supervision for Weakly and Semi-Supervised Semantic Segmentation.

Existing studies on semantic segmentation using image-level weak supervision have several limitations, including sparse object coverage, inaccurate object boundaries, and co-occurring pixels from non-target objects. To overcome these challenges, we propose a novel framework, an improved version of Explicit Pseudo-pixel Supervision (EPS++), which learns from pixel-level feedback by combining two types of weak supervision. Specifically, the image-level label provides the object identity via the localization map, and the saliency map from an off-the-shelf saliency detection model offers rich object boundaries. We devise a joint training strategy to fully utilize the complementary relationship between disparate information. Notably, we suggest an Inconsistent Region Drop (IRD) strategy, which effectively handles errors in saliency maps using fewer hyper-parameters than EPS. Our method can obtain accurate object boundaries and discard co-occurring pixels, significantly improving the quality of pseudo-masks. Experimental results show that EPS++ effectively resolves the key challenges of semantic segmentation using weak supervision, resulting in new state-of-the-art performances on three benchmark datasets in a weakly supervised semantic segmentation setting. Furthermore, we show that the proposed method can be extended to solve the semi-supervised semantic segmentation problem using image-level weak supervision. Surprisingly, the proposed model also achieves new state-of-the-art performances on two popular benchmark datasets.

Learning From Atypical Behavior: Temporary Interest Aware Recommendation Based on Reinforcement Learning

Traditional robust recommendation methods view atypical user-item interactions as noise and aim to reduce their impact with some kind of noise filtering technique, which often suffers from two challenges. First, in real world, atypical interactions may signal users' temporary interest different from their general preference. Therefore, simply filtering out the atypical interactions as noise may be inappropriate and degrade the personalization of recommendations. Second, it is hard to acquire the temporary interest since there are no explicit supervision signals to indicate whether an interaction is atypical or not. To address this challenges, we propose a novel model called Temporary Interest Aware Recommendation (TIARec), which can distinguish atypical interactions from normal ones without supervision and capture the temporary interest as well as the general preference of users. Particularly, we propose a reinforcement learning framework containing a recommender agent and an auxiliary classifier agent, which are jointly trained with the objective of maximizing the cumulative return of the recommendations made by the recommender agent. During the joint training process, the classifier agent can judge whether the interaction with an item recommended by the recommender agent is atypical, and the knowledge about learning temporary interest from atypical interactions can be transferred to the recommender agent, which makes the recommender agent able to alone make recommendations that balance the general preference and temporary interest of users. At last, the experiments conducted on real world datasets verify the effectiveness of TIARec.

Self-supervised depth super-resolution with contrastive multiview pre-training

Many low-level vision tasks, including guided depth super-resolution (GDSR), struggle with the issue of insufficient paired training data. Self-supervised learning is a promising solution, but it remains challenging to upsample depth maps without the explicit supervision of high-resolution target images. To alleviate this problem, we propose a self-supervised depth super-resolution method with contrastive multiview pre-training. Unlike existing contrastive learning methods for classification or segmentation tasks, our strategy can be applied to regression tasks even when trained on a small-scale dataset and can reduce information redundancy by extracting unique features from the guide. Furthermore, we propose a novel mutual modulation scheme that can effectively compute the local spatial correlation between cross-modal features. Exhaustive experiments demonstrate that our method attains superior performance with respect to state-of-the-art GDSR methods and exhibits good generalization to other modalities.

Learning feature fusion via an interpretation method for tumor segmentation on PET/CT

Accurate tumor segmentation of multi-modality PET/CT images plays a vital role in computer-aided cancer diagnosis and treatment. It is crucial to rationally fuse the complementary information in multi-modality PET/CT segmentation. However, existing methods usually lack interpretability and fail to sufficiently identify and aggregate critical information from different modalities. In this study, we proposed a novel segmentation framework that incorporated an interpretation module into the multi-modality segmentation backbone. The interpretation module highlighted critical features from each modality based on their contributions to the segmentation performance. To provide explicit supervision for the interpretation module, we introduced a novel interpretation loss with two fusion schemes: strengthened fusion and perturbed fusion. The interpretation loss guided the interpretation module to focus on informative features, enhancing its effectiveness in generating meaningful interpretable masks. Under the guidance of the interpretation module, the proposed approach can fully exploit meaningful features from each modality, leading to better integration of multi-modality information and improved segmentation performance. Ablative and comparative experiments were conducted on two PET/CT tumor segmentation datasets. The proposed approach surpassed the baseline by 1.4 and 1.8 Dices on two datasets, respectively, indicating the improvement achieved by the interpretation method. Furthermore, the proposed approach outperformed the best comparison approach by 0.9 and 0.6 Dices on two datasets, respectively. In addition, visualization and perturbation experiments further illustrated the effectiveness of the interpretation method in highlighting critical features.

Learning Implicit Neural Representation for Satellite Object Mesh Reconstruction

Constructing a surface representation from the sparse point cloud of a satellite is an important task for satellite on-orbit services such as satellite docking and maintenance. In related studies on surface reconstruction from point clouds, implicit neural representations have gained popularity in learning-based 3D object reconstruction. When aiming for a satellite with a more complicated geometry and larger intra-class variance, existing implicit approaches cannot perform well. To solve the above contradictions and make effective use of implicit neural representations, we built a NASA3D dataset containing point clouds, watertight meshes, occupancy values, and corresponding points by using the 3D models on NASA’s official website. On the basis of NASA3D, we propose a novel network called GONet for a more detailed reconstruction of satellite grids. By designing an explicit-related implicit neural representation of the Grid Occupancy Field (GOF) and introducing it into GONet, we compensate for the lack of explicit supervision in existing point cloud surface reconstruction approaches. The GOF, together with the occupancy field (OF), serves as the supervised information for neural network learning. Learning the GOF strengthens GONet’s attention to the critical points of the surface extraction algorithm Marching Cubes; thus, it helps improve the reconstructed surface’s accuracy. In addition, GONet uses the same encoder and decoder as ConvONet but designs a novel Adaptive Feature Aggregation (AFA) module to achieve an adaptive fusion of planar and volume features. The insertion of AFA allows for the obtained implicit features to incorporate more geometric and volumetric information. Both visualization and quantitative experimental results demonstrate that our GONet could handle 3D satellite reconstruction work and outperform existing state-of-the-art methods by a significant margin. With a watertight mesh, our GONet achieves 5.507 CD-L1, 0.8821 F-score, and 68.86% IoU, which is equal to gains of 1.377, 0.0466, and 3.59% over the previous methods using NASA3D, respectively.

A type-augmented knowledge graph embedding framework for knowledge graph completion

Knowledge graphs (KGs) are of great importance to many artificial intelligence applications, but they usually suffer from the incomplete problem. Knowledge graph embedding (KGE), which aims to represent entities and relations in low-dimensional continuous vector spaces, has been proved to be a promising approach for KG completion. Traditional KGE methods only concentrate on structured triples, while paying less attention to the type information of entities. In fact, incorporating entity types into embedding learning could further improve the performance of KG completion. To this end, we propose a universal Type-augmented Knowledge graph Embedding framework (TaKE) which could utilize type features to enhance any traditional KGE models. TaKE automatically captures type features under no explicit type information supervision. And by learning different type representations of each entity, TaKE could distinguish the diversity of types specific to distinct relations. We also design a new type-constrained negative sampling strategy to construct more effective negative samples for the training process. Extensive experiments on four datasets from three real-world KGs (Freebase, WordNet and YAGO) demonstrate the merits of our proposed framework. In particular, combining TaKE with the recent tensor factorization KGE model SimplE can achieve state-of-the-art performance on the KG completion task.