Joint Learning Research Articles

Dynamic Feature Focusing Network for small object detection

Deep learning has driven research in object detection and achieved proud results. Despite its significant advancements in object detection, small object detection still struggles with low recognition rates and inaccurate positioning, primarily attributable to their miniature size. The location deviation of small objects induces severe feature misalignment, and the disequilibrium between classification and regression tasks hinders accurate recognition. To address these issues, we propose a Dynamic Feature Focusing Network (DFFN), which contains a duo of crucial modules: Visual Perception Enhancement Module (VPEM) and Task Association Module (TAM). Drawing upon the deformable convolution and attention mechanism, the VPEM concentrates on sparse key features and perceives the misalignment via positional offset. We aggregate multi-level features at identical spatial locations via layer average operation for learning a more discriminative representation. Incorporating class alignment and bounding box alignment parts, the TAM promotes classification ability, refines bounding box regression, and facilitates the joint learning of classification and localization. We conduct diverse experiments, and the proposed method considerably enhances the small object detection performance on four benchmark datasets of MS COCO, VisDrone, VOC, and TinyPerson. Our method has improved by 3.4 and 2.2 in mAP and APs, making solid improvements on COCO. Compared to other classic detection models, DFFN exhibits a high level of competitiveness in precision.

Maintenance cost optimisation in critical single-component systems: A technician’s training approach based on a joint learning–forgetting and fuzzy maintenance quality model

Effective maintenance in critical systems is crucial to ensure their operability and safety. Maintenance technicians play an important role in achieving the safety and reliability objectives of critical system components. However, the role of the technical team in maintenance quality is often overlooked. Our study addresses this knowledge gap by developing a methodology to optimise the training scheduling of maintenance technicians in single-component systems, focusing on a learn-and-forget approach. Our main objective is to improve maintenance efficiency by mitigating the negative impact of technician expertise level. To this end, we implemented a simulation-based optimisation, modelling the evolution of technician experience and its influence on maintenance costs. As a case study, we used an urban aerial passenger ropeway system, demonstrating that proper training of technical staff can significantly reduce both corrective and preventive maintenance costs. Our methodology identified three strategic areas for training technical staff throughout the system lifecycle. The results highlight the importance of investing in the training scheduling of technical equipment in critical systems and suggest that an appropriate training strategy can improve operational efficiency and reduce maintenance costs. In conclusion, our research underlines the need to actively consider the human factor in the maintenance of critical systems to ensure their optimal performance and long-term safety.

DMANet: Dual-modality alignment network for visible–infrared person re-identification

Visible–infrared person re-identification (VI-ReID) is a challenging retrieval task, which aims to match the same pedestrian between visible and infrared modalities. Most existing works achieve performance gains by solving the problem of the inherent cross-modality discrepancies. However, they cannot fully mine the modality information and lead to a poor generalization. In addition, the pedestrian images are unable to align well due to the large inter- and intra- class variations. To tackle the above limitations, we propose a novel dual-modality alignment network (DMANet) for VI-ReID. The core idea of our work is to develop multi-granularity features mutual learning (MGFML) for inadequate perception of modalities information, and to solve modality difference by proposing inter- and intra- modality alignment module (IIMA). Specifically, firstly, an effective multi-granularity features mutual learning module is proposed to mine the multi-granularity features, which combines the domain alignment and self-distillation to relieve modality discrepancy. Further, the maximum mean discrepancy loss and mutual learning loss are presented to enhance the identity-aware ability of the DMANet. Secondly, an effective inter- and intra- modality alignment module is presented to explore the potential alignment relation of inter- and intra- modalities. Finally, joint learning mechanism of multi-granularity features and modality alignment is utilized to improve the VI-ReID accuracy. Extensive experiments on mainstream benchmarks demonstrate that our method is superior to the state-of-the-art methods.

Nonfactoid Question Answering as Query-Focused Summarization With Graph-Enhanced Multihop Inference.

Nonfactoid question answering (QA) is one of the most extensive yet challenging applications and research areas in natural language processing (NLP). Existing methods fall short of handling the long-distance and complex semantic relations between the question and the document sentences. In this work, we propose a novel query-focused summarization method, namely a graph-enhanced multihop query-focused summarizer (GMQS), to tackle the nonfactoid QA problem. Specifically, we leverage graph-enhanced reasoning techniques to elaborate the multihop inference process in nonfactoid QA. Three types of graphs with different semantic relations, namely semantic relevance, topical coherence, and coreference linking, are constructed for explicitly capturing the question-document and sentence-sentence interrelationships. Relational graph attention network (RGAT) is then developed to aggregate the multirelational information accordingly. In addition, the proposed method can be adapted to both extractive and abstractive applications as well as be mutually enhanced by joint learning. Experimental results show that the proposed method consistently outperforms both existing extractive and abstractive methods on two nonfactoid QA datasets, WikiHow and PubMedQA, and possesses the capability of performing explainable multihop reasoning.

Joint Participant and Learning Topology Selection for Federated Learning in Edge Clouds

Joint Participant and Learning Topology Selection for Federated Learning in Edge Clouds

Multidomain Object Detection Framework Using Feature Domain Knowledge Distillation.

Object detection techniques have been widely studied, utilized in various works, and have exhibited robust performance on images with sufficient luminance. However, these approaches typically struggle to extract valuable features from low-luminance images, which often exhibit blurriness and dim appearence, leading to detection failures. To overcome this issue, we introduce an innovative unsupervised feature domain knowledge distillation (KD) framework. The proposed framework enhances the generalization capability of neural networks across both low- and high-luminance domains without incurring additional computational costs during testing. This improvement is made possible through the integration of generative adversarial networks and our proposed unsupervised KD process. Furthermore, we introduce a region-based multiscale discriminator designed to discern feature domain discrepancies at the object level rather than from the global context. This bolsters the joint learning process of object detection and feature domain distillation tasks. Both qualitative and quantitative assessments shown that the proposed method, empowered by the region-based multiscale discriminator and the unsupervised feature domain distillation process, can effectively extract beneficial features from low-luminance images, outperforming other state-of-the-art approaches in both low- and sufficient-luminance domains.

Efficient Multi-View Clustering via Unified and Discrete Bipartite Graph Learning.

Although previous graph-based multi-view clustering (MVC) algorithms have gained significant progress, most of them are still faced with three limitations. First, they often suffer from high computational complexity, which restricts their applications in large-scale scenarios. Second, they usually perform graph learning either at the single-view level or at the view-consensus level, but often neglect the possibility of the joint learning of single-view and consensus graphs. Third, many of them rely on the k -means for discretization of the spectral embeddings, which lack the ability to directly learn the graph with discrete cluster structure. In light of this, this article presents an efficient MVC approach via u nified and d iscrete b ipartite g raph l earning (UDBGL). Specifically, the anchor-based subspace learning is incorporated to learn the view-specific bipartite graphs from multiple views, upon which the bipartite graph fusion is leveraged to learn a view-consensus bipartite graph with adaptive weight learning. Furthermore, the Laplacian rank constraint is imposed to ensure that the fused bipartite graph has discrete cluster structures (with a specific number of connected components). By simultaneously formulating the view-specific bipartite graph learning, the view-consensus bipartite graph learning, and the discrete cluster structure learning into a unified objective function, an efficient minimization algorithm is then designed to tackle this optimization problem and directly achieve a discrete clustering solution without requiring additional partitioning, which notably has linear time complexity in data size. Experiments on a variety of multi-view datasets demonstrate the robustness and efficiency of our UDBGL approach. The code is available at https://github.com/huangdonghere/UDBGL.

Matching cities by their climate change measures and blockages profile for mutual-learning.

More and more cities are working on climate resilience by adding climate adaptive measures to their mitigation programs, but choices must be made and barriers must be overcome, as shown by the extensive research in this area, including our own previously conducted exploratory research (Sanders & Oliveira, 2020). The motivation for new follow-up research has been to develop an instrument with which cities can compare each other’s climate programs, to find the right city partner to start a process of joint learning from each other. After studying current research classifying cities' climate adaptive measures, implemented or in planning, most research is into processes to help cities find the right measures, it has been concluded that the level of detail on which this research generally focuses is too detailed to help cities comparing to find their matching city partner. That is why it was ultimately decided to extent our own diagram from the aforementioned exploratory research, which is less abstract, with the aim of providing cities with this improved instrument. It should be noted that given the higher level of detail, we should speak of blockages rather than barriers. The recent results of the H2020 Marie Curie 'SOS Climate Waterfront' study were used to test this improved diagram, as data on climate adaptive measures from six European coastal cities recently became available. A test diagram was then generated for each of these cities and matching has been selected to evaluate the diagram, in order to make a well-functioning instrument available. The final conclusion is that the diagram provides a recognizable matching for the six SOS cities, but that improvement of the diagram is recommended during further use.

“All were gaining knowledge from each other”: decolonial participatory research capacity-sharing for and by non-academics

ABSTRACT This article contributes to the emerging literature on decolonial research capacity-sharing (the process of strengthening individual/organisational capacity to shape research agendas, assess, design, produce, disseminate, and apply evidence). It provides a discussion of a participatory research capacity-sharing initiative led by Joint Learning Initiative on Faith and Local Communities (JLI), which was aimed at practitioners, activists, and researchers, with valuable experience as professionals and activists but little to no prior research experience, from various parts of Africa, Asia, Europe, and the Middle East. While existing literature (including publications on decolonial research capacity-sharing) often assumes that research capacity-sharing is led by and aimed at academics, our article reflects on the question of how decolonial research capacity-sharing can be implemented in interventions led by and aimed at non-academics. It therefore makes an important contribution to existing literature on research capacity-sharing, pointing to the important role practitioners, activists, and community members can play, especially in interventions informed by decolonial principles.

Cross-Modal Retrieval-enhanced code Summarization based on joint learning for retrieval and generation

Context:Code summarization refers to a task that automatically generates a natural language description of a code snippet to facilitate code comprehension. Existing methods have achieved satisfactory results by incorporating information retrieval into generative deep-learning models for reusing summaries of existing code. However, most of these existing methods employed non-learnable generic retrieval methods for content-based retrieval, resulting in a lack of diversity in the retrieved results during training, thereby making the model over-reliant on retrieved results and reducing the generative model’s ability to generalize to unknown samples. Objective:To address this issue, this paper introduces CMR-Sum: a novel Cross-Modal Retrieval-enhanced code Summarization framework based on joint learning for generation and retrieval tasks, where both two tasks are allowed to be optimized simultaneously. Method:Specifically, we use a cross-modal retrieval module to dynamically alter retrieval results during training, which enhances the diversity of the retrieved results and maintains a relative balance between the two tasks. Furthermore, in the summary generation phase, we employ a cross-attention mechanism to generate code summaries based on the alignment between retrieved and generated summaries. We conducted experiments on three real-world datasets, comparing the performance of our method with baseline models. Additionally, we performed extensive qualitative analysis. Result:Results from qualitative and quantitative experiments indicate that our approach effectively enhances the performance of code summarization. Our method outperforms both the generation-based and the retrieval-enhanced baselines. Further ablation experiments demonstrate the effectiveness of each component of our method. Results from sensitivity analysis experiments suggest that our approach achieves good performance without requiring extensive hyper-parameter search. Conclusion:The direction of utilizing retrieval-enhanced generation tasks shows great potential. It is essential to increase the diversity of retrieval results during the training process, which is crucial for improving the generality and the performance of the model.

Leveraging segmentation-guided spatial feature embedding for overall survival prediction in glioblastoma with multimodal magnetic resonance imaging

Background and objectivePatients with glioblastoma have a five-year relative survival rate of less than 5 %. Thus, accurately predicting the overall survival (OS) of patients with glioblastoma is crucial for effective treatment planning. MethodsTo fully leverage the imaging characteristics of glioblastomas, we propose a segmentation-guided regression method for predicting OS of patients with brain tumors using multimodal magnetic resonance imaging. Specifically, a brain tumor segmentation network was first pre-trained without leveraging survival information. Subsequently, the survival regression network was jointly trained with the guidance of brain tumor segmentation, focusing on tumor voxels and suppressing irrelevant backgrounds. ResultsOur proposed framework, based on the well-known backbone of UNETR++, achieved a Dice score of 0.7910, Spearman correlation of 0.4112, and Harrell's concordance index of 0.6488. The model consistently showed promising results compared with baseline methods on two different datasets (BraTS and UCSF-PDGM). Furthermore, ablation studies on our training configurations demonstrated that both the pre-training segmentation network and contrastive loss significantly improved all metrics for OS prediction. ConclusionsIn this study, we propose a joint learning framework based on a pre-trained segmentation backbone for OS prediction by leveraging a brain tumor segmentation map. By utilizing a spatial feature map, our model can operate using a sliding-window approach, which can be adopted by varying the matrix sizes and resolutions of the input images.

TGGS network: A multi-task learning network for gradient-guided knowledge sharing

Multi-task learning (MTL) has been widely used in various fields, such as time series data prediction and image classification. Most existing deep MTL methods achieve joint learning by sharing common knowledge with task relevance. However, they rely solely on self-exploration to mine common knowledge across the tasks, which is unguided and purposeless. Unlike these methods, in this paper, we propose a novel Task Gradient Guided Sharing (TGGS) MTL network for purposefully guiding the model to share knowledge. Specifically, we define a confidence matrix for each task for selecting effective knowledge from the common knowledge to share. To realize an effective guidance, we align the confidence matrix with a class gradient mapping (CGM) computed based on the task-specific gradients. Consequently, our method effectively employs task-specific gradients to guide the model in sharing useful knowledge. More importantly, the confidence matrix explains the complex correlations among tasks in MTL. As our method exclusively focuses on guided knowledge sharing, the TGGS network can seamlessly integrate with diverse MTL sharing strategies. We conduct extensive experiments on time series datasets and image datasets. The experimental results demonstrate our method significantly improves the performance of these MTL strategies.

JointNet: Multitask Learning Framework for Denoising and Detecting Anomalies in Hyperspectral Remote Sensing

One of the significant challenges with traditional single-task learning-based anomaly detection using noisy hyperspectral images (HSIs) is the loss of anomaly targets during denoising, especially when the noise and anomaly targets are similar. This issue significantly affects the detection accuracy. To address this problem, this paper proposes a multitask learning (MTL)-based method for detecting anomalies in noisy HSIs. Firstly, a preliminary detection approach based on the JointNet model, which decomposes the noisy HSI into a pure background and a noise–anomaly target mixing component, is introduced. This approach integrates the minimum noise fraction rotation (MNF) algorithm into an autoencoder (AE), effectively isolating the noise while retaining critical features for anomaly detection. Building upon this, the JointNet model is further optimized to ensure that the noise information is shared between the denoising and anomaly detection subtasks, preserving the integrity of the training data during the anomaly detection process and resolving the issue of losing anomaly targets during denoising. A novel loss function is designed to enable the joint learning of both subtasks under the multitask learning model. In addition, a noise score evaluation metric is introduced to calculate the probability of a pixel being an anomaly target, allowing for a clear distinction between noise and anomaly targets, thus providing the final anomaly detection results. The effectiveness of the proposed model and method is validated via testing on the HYDICE and San Diego datasets. The denoising metric results of the PSNR, SSIM, and SAM are 41.79, 0.91, and 4.350 and 42.83, 0.93, and 3.558 on the HYDICE and San Diego datasets, respectively. The anomaly detection ACU is 0.943 and 0.959, respectively. The proposed method outperforms the other algorithms, demonstrating that the reconstructed images using this method exhibited lower noise levels and more complete image information, and the JointNet model outperforms the mainstream HSI anomaly detection algorithms in both the quantitative evaluation and visual effect, showcasing its improved detection capabilities.

B-MAR: bidirectional artifact representations learning framework for metal artifact reduction in dental CBCT

Objective. Metal artifacts in computed tomography (CT) images hinder diagnosis and treatment significantly. Specifically, dental cone-beam computed tomography (Dental CBCT) images are seriously contaminated by metal artifacts due to the widespread use of low tube voltages and the presence of various high-attenuation materials in dental structures. Existing supervised metal artifact reduction (MAR) methods mainly learn the mapping of artifact-affected images to clean images, while ignoring the modeling of the metal artifact generation process. Therefore, we propose the bidirectional artifact representations learning framework to adaptively encode metal artifacts caused by various dental implants and model the generation and elimination of metal artifacts, thereby improving MAR performance. Approach. Specifically, we introduce an efficient artifact encoder to extract multi-scale representations of metal artifacts from artifact-affected images. These extracted metal artifact representations are then bidirectionally embedded into both the metal artifact generator and the metal artifact eliminator, which can simultaneously improve the performance of artifact removal and artifact generation. The artifact eliminator learns artifact removal in a supervised manner, while the artifact generator learns artifact generation in an adversarial manner. To further improve the performance of the bidirectional task networks, we propose artifact consistency loss to align the consistency of images generated by the eliminator and the generator with or without embedding artifact representations. Main results. To validate the effectiveness of our algorithm, experiments are conducted on simulated and clinical datasets containing various dental metal morphologies. Quantitative metrics are calculated to evaluate the results of the simulation tests, which demonstrate b-MAR improvements of >1.4131 dB in PSNR, >0.3473 HU decrements in RMSE, and >0.0025 promotion in structural similarity index measurement over the current state-of-the-art MAR methods. All results indicate that the proposed b-MAR method can remove artifacts caused by various metal morphologies and restore the structural integrity of dental tissues effectively. Significance. The proposed b-MAR method strengthens the joint learning of the artifact removal process and the artifact generation process by bidirectionally embedding artifact representations, thereby improving the model’s artifact removal performance. Compared with other comparison methods, b-MAR can robustly and effectively correct metal artifacts in dental CBCT images caused by different dental metals.

TAJ-Net: a two-stage clustered cell segmentation network with adaptive joint learning of spatial and spectral information.

Pulmonary adenocarcinoma is the primary cause of cancer-related death worldwide and pathological diagnosis is the "golden standard" based on the regional distribution of cells. Thus, regional cell segmentation is a key step while it is challenging due to the following reasons: 1) It is hard for pure semantic and instance segmentation methods to obtain a high-quality regional cell segmentation result; 2) Since the spatial appearances of pulmonary cells are very similar which even confuse pathologists, annotation errors are usually inevitable. Considering these challenges, we propose a two-stage 3D adaptive joint training framework (TAJ-Net) to segment-then-classify cells with extra spectral information as the supplementary information of spatial information. Firstly, we propose to leverage a few-shot method with limited data for cell mask acquisition to avoid the disturbance of cluttered backgrounds. Secondly, we introduce an adaptive joint training strategy to remove noisy samples through two 3D networks and one 1D network for cell type classification rather than segmentation. Subsequently, we propose a patch mapping method to map classification results to the original images to obtain regional segmentation results. In order to verify the effectiveness of TAJ-Net, we build two 3D hyperspectral datasets, i.e., pulmonary adenocarcinoma (3,660 images) and thyroid carcinoma (4623 images) with 40 bands. The first dataset will be released for further research. Experiments show that TAJ-Net achieves much better performance in clustered cell segmentation, and it can regionally segment different kinds of cells with high overlap and blurred edges, which is a difficult task for the state-of-the-art methods. Compared to 2D models, the hyperspectral image-based 3D model reports a significant improvement of up to 11.5% in terms of the Dice similarity coefficient in the pulmonary adenocarcinoma dataset.

Deep learning in autonomous driving: Advantages, limitations, and innovative solutions

With the rapid development of autonomous driving technology, deep learning has become a core driver for innovation in testing autonomous driving scenarios. This review paper delves into the critical role of deep learning in autonomous driving technology. The paper will describe how deep learning is at the center of driving innovation. The paper thoroughly explores the application of deep learning in obstacle detection, scene classification and understanding, and image segmentation, emphasizing the significant benefits in perception and decision-making while pointing out the challenges and innovative solutions adopted. The innovative solutions section proposes multimodal fusion and joint learning, new methods for 3D semantic segmentation, etc., aiming to improve image segmentation's accuracy and generalization ability. Overall, deep learning has great potential in automated driving technology, and by innovating and solving challenges, it will advance the system and provide reliable, intelligent, and efficient solutions for future transportation systems.

Instance Consistency Regularization for Semi-Supervised 3D Instance Segmentation.

Large-scale datasets with point-wise semantic and instance labels are crucial to 3D instance segmentation but also expensive. To leverage unlabeled data, previous semi-supervised 3D instance segmentation approaches have explored self-training frameworks, which rely on high-quality pseudo labels for consistency regularization. They intuitively utilize both instance and semantic pseudo labels in a joint learning manner. However, semantic pseudo labels contain numerous noise derived from the imbalanced category distribution and natural confusion of similar but distinct categories, which leads to severe collapses in self-training. Motivated by the observation that 3D instances are non-overlapping and spatially separable, we ask whether we can solely rely on instance consistency regularization for improved semi-supervised segmentation. To this end, we propose a novel self-training network InsTeacher3D to explore and exploit pure instance knowledge from unlabeled data. We first build a parallel base 3D instance segmentation model DKNet, which distinguishes each instance from the others via discriminative instance kernels without reliance on semantic segmentation. Based on DKNet, we further design a novel instance consistency regularization framework to generate and leverage high-quality instance pseudo labels. Experimental results on multiple large-scale datasets show that the InsTeacher3D significantly outperforms prior state-of-the-art semi-supervised approaches.

Multi-Task Learning of Object States and State-Modifying Actions From Web Videos.

We aim to learn to temporally localize object state changes and the corresponding state-modifying actions by observing people interacting with objects in long uncurated web videos. We introduce three principal contributions. First, we develop a self-supervised model for jointly learning state-modifying actions together with the corresponding object states from an uncurated set of videos from the Internet. The model is self-supervised by the causal ordering signal, i.e., initial object state → manipulating action → end state. Second, we explore alternative multi-task network architectures and identify a model that enables efficient joint learning of multiple object states and actions, such as pouring water and pouring coffee, together. Third, we collect a new dataset, named ChangeIt, with more than 2600 hours of video and 34 thousand changes of object states. We report results on an existing instructional video dataset COIN as well as our new large-scale ChangeIt dataset containing tens of thousands of long uncurated web videos depicting various interactions such as hole drilling, cream whisking, or paper plane folding. We show that our multi-task model achieves a relative improvement of 40% over the prior methods and significantly outperforms both image-based and video-based zero-shot models for this problem.

A Joint Learning Recommendation Model for E-Commerce Platforms Integrating Long-Term and Short-Term Interests

A Joint Learning Recommendation Model for E-Commerce Platforms Integrating Long-Term and Short-Term Interests

SnO2 induced electrostatic polarization PVDF composite nanofibers for efficient energy harvesting and self-powered wireless monitoring /motion recognition systems

With the rapid development of the Internet of Things and flexible electronics, piezoelectric nanogenerators (PENGs) based on polyvinylidene fluoride (PVDF) have demonstrated a wide range of potential applications in wearable devices. However, there are still challenges in developing high-performance PENGs, which affect their potential applications in more scenarios. Herein, we propose a SnO2-PVDF (SP)-PENG based on low-cost and environment-friendly SnO2 nanoparticles modulated via electrospinning, achieving outstanding piezoelectric output performance. The piezoelectric output of SP-PENG is up to 49.2 V, which is 11.7 times higher than that of pure PVDF-PENG. From the theoretical perspective, the enhanced electrostatic polarization is achieved by the strengthened local electric field due to the presence of low-resistivity SnO2 in the SP fibers. The mechanism of enhanced electrostatic polarization is also declared by COMSOL simulation. Moreover, the SP-PENG exhibits excellent durability (41,600 cycles) and long-term stability (8 months). A self-powered wireless sensing-monitoring system is realized by combining SP-PENG with circuit design. The motion recognition system is accomplished by integrating an assistance of a 1D CNN-LSTM joint learning model, which is verified by an alphabetic handwriting recognition with a classification accuracy up to 100 %. This study provides significant insights for enhancing the performance of PENGs and offers valuable guidance for exploring the application of PENGs in flexible electronics and artificial intelligence.