Multi-task Network Research Articles

Detection of lanes, obstacles and drivable areas for self-driving cars using multifusion perception metrics

<p>Autonomous vehicles have been a recent trend and active research area from the onset of machine learning and deep learning algorithms. Computer vision and deep learning techniques have simplified the operations of continuous monitoring and decision-making capabilities of autonomous vehicles. A navigation system is facilitated by a visual system, where sensors and collectors process input in form of images or videos, and the navigation system will be making certain decisions to adhere to the safety of drivers and passers-by. This research article contemplates the model of obstacle detection, lane detection, and how the vehicle is supposed to act in terms of autonomous driving situation. This situation should resemble human driving conditions and should ensure maximum safety to both the stakeholders. A unified neural network for detecting lanes, objects, obstacles and to advise the driving speed is defined in this architecture. As far as autonomous driving is considered, these target elements are considered to be the predominant areas of focus for autonomous driving vehicles. Since capturing the images or videos have to be performed in real-time scenarios and processing them for relevant decision making have to be completed at a swift pace, a concept of context tensors is introduced in the decoders for discriminating the tasks based on priority. Every task is associated with the other tasks and also the decision-making process and hence this architecture will continue to learn every day. From the obtained results, it is evident that multitask networks can be improved using the proposed method in terms of accuracy, decision-making capability and reduced computational time. This model investigates the performance using Berkeley deep drive datasets which are considered to be a challenging dataset.</p>

DeepmdQCT: A multitask network with domain invariant features and comprehensive attention mechanism for quantitative computer tomography diagnosis of osteoporosis

In the medical field, the application of machine learning technology in the automatic diagnosis and monitoring of osteoporosis often faces challenges related to domain adaptation in drug therapy research. The existing neural networks used for the diagnosis of osteoporosis may experience a decrease in model performance when applied to new data domains due to changes in radiation dose and equipment. To address this issue, in this study, we propose a new method for multi domain diagnostic and quantitative computed tomography (QCT) images, called DeepmdQCT. This method adopts a domain invariant feature strategy and integrates a comprehensive attention mechanism to guide the fusion of global and local features, effectively improving the diagnostic performance of multi domain CT images. We conducted experimental evaluations on a self-created OQCT dataset, and the results showed that for dose domain images, the average accuracy reached 91%, while for device domain images, the accuracy reached 90.5%. our method successfully estimated bone density values, with a fit of 0.95 to the gold standard. Our method not only achieved high accuracy in CT images in the dose and equipment fields, but also successfully estimated key bone density values, which is crucial for evaluating the effectiveness of osteoporosis drug treatment. In addition, we validated the effectiveness of our architecture in feature extraction using three publicly available datasets. We also encourage the application of the DeepmdQCT method to a wider range of medical image analysis fields to improve the performance of multi-domain images.

A Multitask Network for Joint Multispectral Pansharpening on Diverse Satellite Data.

Despite the rapid advance in multispectral (MS) pansharpening, existing convolutional neural network (CNN)-based methods require training on separate CNNs for different satellite datasets. However, such a single-task learning (STL) paradigm often leads to overlooking any underlying correlations between datasets. Aiming at this challenging problem, a multitask network (MTNet) is presented to accomplish joint MS pansharpening in a unified framework for images acquired by different satellites. Particularly, the pansharpening process of each satellite is treated as a specific task, while MTNet simultaneously learns from all data obtained from these satellites following the multitask learning (MTL) paradigm. MTNet shares the generic knowledge between datasets via task-agnostic subnetwork (TASNet), utilizing task-specific subnetworks (TSSNets) to facilitate the adaptation of such knowledge to a certain satellite. To tackle the limitation of the local connectivity property of the CNN, TASNet incorporates Transformer modules to derive global information. In addition, band-aware dynamic convolutions (BDConvs) are proposed that can accommodate various ground scenes and bands by adjusting their respective receptive field (RF) size. Systematic experimental results over different datasets demonstrate that the proposed approach outperforms the existing state-of-the-art (SOTA) techniques.

A Multi-Task Network With Dynamic Segmentation for Sea Ice Classification in Arctic Shipping Route Optimization

A Multi-Task Network With Dynamic Segmentation for Sea Ice Classification in Arctic Shipping Route Optimization

A deep learning method for multi-task intelligent detection of oral cancer based on optical fiber Raman spectroscopy.

In the fight against oral cancer, innovative methods like Raman spectroscopy and deep learning have become powerful tools, particularly in integral tasks encompassing tumor staging, lymph node staging, and histological grading. These aspects are essential for the development of effective treatment strategies and prognostic assessment. However, it is important to note that most research so far has focused on solutions to one of these problems and has not taken full advantage of the potential wealth of information in the data. To compensate for this shortfall, we conceived a method that combines Raman spectroscopy with deep learning for simultaneous processing of multiple classification tasks, including tumor staging, lymph node staging, and histological grading. To achieve this innovative approach, we collected 1750 Raman spectra from 70 tissue samples, including normal and cancerous tissue samples from 35 patients with oral cancer. In addition, we used a deep neural network architecture to design four distinct multi-task network (MTN) models for intelligent oral cancer diagnosis, named MTN-Alexnet, MTN-Googlenet, MTN-Resnet50, and MTN-Transformer. To determine their effectiveness, we compared these multitask models to each other and to single-task models and traditional machine learning methods. The preliminary experimental results show that our multi-task network model has good performance, among which MTN-Transformer performs best. Specifically, MTN-Transformer has an accuracy of 81.5%, a precision of 82.1%, a sensitivity of 80.2%, and an F1_score of 81.1% in terms of tumor staging. In the field of lymph node staging, the accuracy, precision, sensitivity, and F1_score of MTN-Transformer are 81.3%, 83.0%, 80.1%, and 81.5% respectively. Similarly, for the histological grading classification tasks, the accuracy was 83.0%, the precision 84.3%, the sensitivity 76.7%, and the F1_score 80.2%. This code is available at https://github.com/ISCLab-Bistu/MultiTask-OralRamanSystem.

A Multitask Network for Multiview Stereo Reconstruction: When Semantic Consistency-Based Clustering Meets Depth Estimation Optimization

A Multitask Network for Multiview Stereo Reconstruction: When Semantic Consistency-Based Clustering Meets Depth Estimation Optimization

Adaptive Self-Paced Collaborative and 3-D Adversarial Multitask Network for Semantic Change Detection Using Zhuhai-1 Orbita Hyperspectral Remote Sensing Imagery

Adaptive Self-Paced Collaborative and 3-D Adversarial Multitask Network for Semantic Change Detection Using Zhuhai-1 Orbita Hyperspectral Remote Sensing Imagery

SeisParaNet: A Novel Multitask Network for Seismic Source Characterization in Earthquake Early Warning

SeisParaNet: A Novel Multitask Network for Seismic Source Characterization in Earthquake Early Warning

An Edge-Aware Multitask Network Based on CNN and Transformer Backbone for Farmland Instance Segmentation

An Edge-Aware Multitask Network Based on CNN and Transformer Backbone for Farmland Instance Segmentation

Simultaneous segmentation and classification of colon cancer polyp images using a dual branch multi-task learning network.

Accurate classification and segmentation of polyps are two important tasks in the diagnosis and treatment of colorectal cancers. Existing models perform segmentation and classification separately and do not fully make use of the correlation between the two tasks. Furthermore, polyps exhibit random regions and varying shapes and sizes, and they often share similar boundaries and backgrounds. However, existing models fail to consider these factors and thus are not robust because of their inherent limitations. To address these issues, we developed a multi-task network that performs both segmentation and classification simultaneously and can cope with the aforementioned factors effectively. Our proposed network possesses a dual-branch structure, comprising a transformer branch and a convolutional neural network (CNN) branch. This approach enhances local details within the global representation, improving both local feature awareness and global contextual understanding, thus contributing to the improved preservation of polyp-related information. Additionally, we have designed a feature interaction module (FIM) aimed at bridging the semantic gap between the two branches and facilitating the integration of diverse semantic information from both branches. This integration enables the full capture of global context information and local details related to polyps. To prevent the loss of edge detail information crucial for polyp identification, we have introduced a reverse attention boundary enhancement (RABE) module to gradually enhance edge structures and detailed information within polyp regions. Finally, we conducted extensive experiments on five publicly available datasets to evaluate the performance of our method in both polyp segmentation and classification tasks. The experimental results confirm that our proposed method outperforms other state-of-the-art methods.

GDMNet: A Unified Multi-Task Network for Panoptic Driving Perception

GDMNet: A Unified Multi-Task Network for Panoptic Driving Perception

A Multi-Task Transformer with Local-Global Feature Interaction and Multiple Tumoral Region Guidance for Breast Cancer Diagnosis.

Breast cancer, as a malignant tumor disease, has maintained high incidence and mortality rates over the years. Ultrasonography is one of the primary methods for diagnosing early-stage breast cancer. However, correctly interpreting breast ultrasound images requires massive time from physicians with specialized knowledge and extensive experience. Recently, deep learning-based method have made significant advancements in breast tumor segmentation and classification due to their powerful fitting capabilities. However, most existing methods focus on performing one of these tasks separately, and often failing to effectively leverage information from specific tumor-related areas that hold considerable diagnostic value. In this study, we propose a multi-task network with local-global feature interaction and multiple tumoral region guidance for breast ultrasound-based tumor segmentation and classification. Specifically, we construct a dual-stream encoder, paralleling CNN and Transformer, to facilitate hierarchical interaction and fusion of local and global features. This architecture enables each stream to capitalize on the strengths of the other while preserving its unique characteristics. Moreover, we design a multi-tumoral region guidance module to explicitly learn long-range non-local dependencies within intra-tumoral and peri-tumoral regions from spatial domain, thus providing interpretable cues beneficial for classification. Experimental results on two breast ultrasound datasets show that our network outperforms state-of-the-art methods in tumor segmentation and classification tasks. Compared with the second-best competitive method, our network improves the diagnosis accuracy from 73.64% to 80.21% on a large external validation dataset, which demonstrates its superior generalization capability.

UMT-Net: A Uniform Multi-Task Network With Adaptive Task Weighting

UMT-Net: A Uniform Multi-Task Network With Adaptive Task Weighting

PARFormer: Transformer-Based Multi-Task Network for Pedestrian Attribute Recognition

Pedestrian attribute recognition (PAR) has received increasing attention because of its wide application in video surveillance and pedestrian analysis. Extracting robust feature representation is one of the key challenges in this task. The existing methods primarily rely on convolutional neural networks (CNNs) as the backbone network for feature extraction. However, these methods mainly focus on small discriminative regions while ignoring the global perspective. To overcome these limitations, we propose PARFormer, a pure transformer-based multi-task PAR network consisting of four modules. In the feature extraction module, we build a transformer-based strong baseline for feature extraction, which achieves competitive results on several PAR benchmarks compared with the existing CNN-based baseline methods. Since the PAR task is vulnerable to environmental factors, we enhance feature robustness in the feature processing module and propose an effective data augmentation strategy named batch random mask (BRM) block to reinforce the attentive feature learning of random patches. Furthermore, we propose a multi-attribute center loss (MACL) to augment the inter-attribute discriminability of feature representations. As viewpoints can affect some specific attributes, in the viewpoint perception module, we propose a multi-view contrastive loss (MVCL) that enables the network to exploit the viewpoint information. In the attribute recognition module, we alleviate the negative-positive imbalance problem to generate the attribute predictions. These modules interact and jointly learn a highly discriminative feature space and supervise the generation of the final features. Extensive experimental results show that the proposed PARFormer network performs well compared to the state-of-the-art methods on several public datasets, including PETA, RAP, and PA100K. Code will be released at https: //github.com/xwf199/PARFormer.

Attention Guided Multi-Task Network for Joint CFO and Channel Estimation in OFDM Systems

The existence of high carrier frequency offset (CFO) and fading channels degrades the performance of communication systems in high mobility environments significantly. To address these challenges, in this paper, we propose an attention guided multi-task network for joint CFO and channel estimation in orthogonal frequency division multiplexing (OFDM) systems. Specifically, considering the correlation between CFO estimation and channel estimation, we construct a multi-task neural network framework which consists of a shared branch (SB), a CFO estimation branch (CFOEB) and a channel estimation branch (CEB). Among them, the SB extracts common features of two tasks, and the CFOEB and CEB make more detailed CFO and channel estimation, respectively. To further suppress the influence of CFO estimation error on channel estimation, we introduce an attention guided module (AGM) which completes the re-calibration of the channel dimension on the basis of different CFO characteristics. We train the whole network by an end-to-end paradigm and construct a suitable multi-task loss function to balance the losses between different tasks. Simulation results demonstrate that the proposed attention guided multi-task learning based joint CFO and channel estimation scheme outperforms the conventional methods and has greater robustness under various conditions.

QoE-based Semantic-Aware Resource Allocation for Multi-Task Networks

QoE-based Semantic-Aware Resource Allocation for Multi-Task Networks

Foodnet: multi-scale and label dependency learning-based multi-task network for food and ingredient recognition

Foodnet: multi-scale and label dependency learning-based multi-task network for food and ingredient recognition

Mobip: a lightweight model for driving perception using MobileNet.

The visual perception model is critical to autonomous driving systems. It provides the information necessary for self-driving cars to make decisions in traffic scenes. We propose a lightweight multi-task network (Mobip) to simultaneously perform traffic object detection, drivable area segmentation, and lane line detection. The network consists of a shared encoder for feature extraction and two decoders for handling detection and segmentation tasks collectively. By using MobileNetV2 as the backbone and an extremely efficient multi-task architecture to implement the perception model, our network has great advantages in inference speed. The performance of the multi-task network is verified on a challenging public Berkeley Deep Drive(BDD100K) dataset. The model achieves an inference speed of 58 FPS on NVIDIA Tesla V100 while still maintaining competitive performance on all three tasks compared to other multi-task networks. Besides, the effectiveness and efficiency of the multi-task architecture are verified via ablative studies.

Optical Flow Estimation in Dense Foggy Scenes With Domain-Adaptive Networks

Estimating optical flow in dense foggy scenes is a challenging task. The basic assumptions for computing flow such as brightness and gradient constancy become invalid. To address the problem, we introduce a semi-supervised deep learning method that can learn from real fog images without requiring the corresponding optical flow ground-truths. Our method is a multi-task network, integrating the domain transformation and optical flow networks in one framework. The domain transformation is performed between foggy and clean domains. Under our semi-supervised training strategy, we train our method in a supervised manner with a pair of synthetic fog images, their corresponding clean images and optical flow ground-truths. Subsequently, given a pair of real fog images and a pair of clean images that are not corresponding to each other (unpaired), in the next training batch, we train our network in an unsupervised manner. The supervised and unsupervised training processes are alternated iteratively. Since our method relies on unsupervised learning for real data, we show that it can be used for test-time training. We show in our experiments that performing test-time training improves the results further on our test data. Extensive experiments show that our method outperforms the state-of-the-art methods in estimating optical flow in dense foggy scenes.

Joint Learning of Image Deblurring and Depth Estimation Through Adversarial Multi-Task Network

Self-supervised monocular depth estimation methods have achieved remarkable results on natural clear images. However, it is still a serious challenge to directly recover depth information from blurred images caused by long-time exposure while camera fast moving. To address this issue, we propose a unified framework for simultaneous deblurring and depth estimation (SDDE), which has higher coupling performance and flexibility compared with the simple concatenation strategy of deblurring model and depth estimation model. This framework mainly benefits from three features: (1) a novel Task-aware Fusion Module (TFM) to adaptively select the most relevant intermediate shared features for the dual decoder network by aggregating multi-scale features, (2) a unique Spatial Interaction Module (SIM) to learn higher-order representation in the encoder stage to better describe complex boundaries of different classes in high-dimensional space, and focuses on the task-related region by modeling the pairwise spatial correlation of the holistic tensor, (3) a Priors-Based Composite Regularization term to jointly optimize the shared encoder-dual decoder network. This work was evaluated on multiple datasets, including: Stereo blur [1], KITTI [2],NYUv2 [3], REDS [4] and our own large-scale stereo blur dataset, resulting in state-of-the-art results for depth estimation and image deblurring, respectively.