Task-specific Modules Research Articles

EFNet: A multitask deep learning network for simultaneous quantification of left ventricle structure and function

Purpose:The purpose of this study is to develop an automated method using deep learning for the reliable and precise quantification of left ventricle structure and function from echocardiogram videos, eliminating the need to identify end-systolic and end-diastolic frames. This addresses the variability and potential inaccuracies associated with manual quantification, aiming to improve the diagnosis and management of cardiovascular conditions. Methods:A single, fully automated multitask network, the EchoFused Network (EFNet) is introduced that simultaneously addresses both left ventricle segmentation and ejection fraction estimation tasks through cross-module fusion. Our proposed approach utilizes semi-supervised learning to estimate the ejection fraction from the entire cardiac cycle, yielding more dependable estimations and obviating the need to identify specific frames. To facilitate joint optimization, the losses from task-specific modules are combined using a normalization technique, ensuring commensurability on a comparable scale. Results:The assessment of the proposed model on a publicly available dataset, EchoNet-Dynamic, shows significant performance improvement, achieving an MAE of 4.35% for ejection fraction estimation and DSC values of 0.9309 (end-diastolic) and 0.9135 (end-systolic) for left ventricle segmentation. Conclusions:The study demonstrates the efficacy of EFNet, a multitask deep learning network, in simultaneously quantifying left ventricle structure and function through cross-module fusion on echocardiogram data.

MFOS: Model-Free & One-Shot Object Pose Estimation

Existing learning-based methods for object pose estimation in RGB images are mostly model-specific or category based. They lack the capability to generalize to new object categories at test time, hence severely hindering their practicability and scalability. Notably, recent attempts have been made to solve this issue, but they still require accurate 3D data of the object surface at both train and test time. In this paper, we introduce a novel approach that can estimate in a single forward pass the pose of objects never seen during training, given minimum input. In contrast to existing state-of-the-art approaches, which rely on task-specific modules, our proposed model is entirely based on a transformer architecture, which can benefit from recently proposed 3D-geometry general pretraining. We conduct extensive experiments and report state-of-the-art one-shot performance on the challenging LINEMOD benchmark. Finally, extensive ablations allow us to determine good practices with this relatively new type of architecture in the field.

VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding

Large-scale pre-trained models have achieved remarkable success in various computer vision tasks. A standard approach to leverage these models is to fine-tune all model parameters for downstream tasks, which poses challenges in terms of computational and storage costs. Recently, inspired by Natural Language Processing (NLP), parameter-efficient transfer learning has been successfully applied to vision tasks. However, most existing techniques primarily focus on single-task adaptation, and despite limited research on multi-task adaptation, these methods often exhibit suboptimal training/inference efficiency. In this paper, we first propose an once-for-all Vision Multi-Task Adapter (VMT-Adapter), which strikes approximately O(1) training and inference efficiency w.r.t task number. Concretely, VMT-Adapter shares the knowledge from multiple tasks to enhance cross-task interaction while preserves task-specific knowledge via independent knowledge extraction modules. Notably, since task-specific modules require few parameters, VMT-Adapter can handle an arbitrary number of tasks with a negligible increase of trainable parameters. We also propose VMT-Adapter-Lite, which further reduces the trainable parameters by learning shared parameters between down- and up-projections. Extensive experiments on four dense scene understanding tasks demonstrate the superiority of VMT-Adapter(-Lite), achieving a 3.96% (1.34%) relative improvement compared to single-task full fine-tuning, while utilizing merely ～1% (0.36%) trainable parameters of the pre-trained model.

OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue

AbstractThis paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: Dialogue state tracker (DST) and response generator (RG). The dialogue state consists of the domain-slot-value triples, which are regarded as the user’s constraints to search the domain-related databases. The large-scale task-oriented dialogue data with the annotated structured dialogue state usually are inaccessible. It prevents the development of the pretrained language model for the task-oriented dialogue. We propose a simple yet effective pretraining method to alleviate this problem, which consists of two pretraining phases. The first phase is to pretrain on large-scale contextual text data, where the structured information of the text is extracted by the information extracting tool. To bridge the gap between the pretraining method and downstream tasks, we design two pretraining tasks: ontology-like triple recovery and next-text generation, which simulates the DST and RG, respectively. The second phase is to fine-tune the pretrained model on the TOD data. The experimental results show that our proposed method achieves an exciting boost and obtains competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks.

Communication-Efficient Federated Multitask Learning Over Wireless Networks

This article investigates the scheduling framework of the federated multitask learning (FMTL) problem with a hard-cooperation structure over wireless networks, in which the scheduling becomes more challenging due to the different convergence behaviors of different tasks. Based on the special model structure, we propose a dynamic user and task scheduling scheme with a block-wise incremental gradient aggregation algorithm, in which the neural network model is decomposed into a common feature-extraction module and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> task-specific modules. Different block gradients with respect to different modules can be scheduled separately. We further propose a Lyapunov-drift-based scheduling scheme that minimizes the overall communication latency by utilizing both the instantaneous data importance and the channel state information. We prove that the proposed scheme can converge almost surely to a KKT solution of the training problem such that the data-distortion issue is resolved. Simulation results illustrate that the proposed scheme significantly reduces the communication latency compared to the state-of-the-art baseline schemes.

Disentangling Light Fields for Super-Resolution and Disparity Estimation.

Light field (LF) cameras record both intensity and directions of light rays, and encode 3D scenes into 4D LF images. Recently, many convolutional neural networks (CNNs) have been proposed for various LF image processing tasks. However, it is challenging for CNNs to effectively process LF images since the spatial and angular information are highly inter-twined with varying disparities. In this paper, we propose a generic mechanism to disentangle these coupled information for LF image processing. Specifically, we first design a class of domain-specific convolutions to disentangle LFs from different dimensions, and then leverage these disentangled features by designing task-specific modules. Our disentangling mechanism can well incorporate the LF structure prior and effectively handle 4D LF data. Based on the proposed mechanism, we develop three networks (i.e., DistgSSR, DistgASR and DistgDisp) for spatial super-resolution, angular super-resolution and disparity estimation. Experimental results show that our networks achieve state-of-the-art performance on all these three tasks, which demonstrates the effectiveness, efficiency, and generality of our disentangling mechanism. Project page: https://yingqianwang.github.io/DistgLF/.

Multi-Feature Super-Resolution Network for Cloth Wrinkle Synthesis

Existing physical cloth simulators suffer from expensive computation and difficulties in tuning mechanical parameters to get desired wrinkling behaviors. Data-driven methods provide an alternative solution. It typically synthesizes cloth animation at a much lower computational cost, and also creates wrinkling effects that highly resemble the much controllable training data. In this paper we propose a deep learning based method for synthesizing cloth animation with high resolution meshes. To do this we first create a dataset for training: a pair of low and high resolution meshes are simulated and their motions are synchronized. As a result the two meshes exhibit similar large-scale deformation but different small wrinkles. Each simulated mesh pair are then converted into a pair of low and high resolution "images" (a 2D array of samples), with each sample can be interpreted as any of three features: the displacement, the normal and the velocity. With these image pairs, we design a multi-feature super-resolution (MFSR) network that jointly train an upsampling synthesizer for the three features. The MFSR architecture consists of two key components: a sharing module that takes multiple features as input to learn low-level representations from corresponding super-resolution tasks simultaneously; and task-specific modules focusing on various high-level semantics. Frame-to-frame consistency is well maintained thanks to the proposed kinematics-based loss function. Our method achieves realistic results at high frame rates: 12-14 times faster than traditional physical simulation. We demonstrate the performance of our method with various experimental scenes, including a dressed character with sophisticated collisions.

System Modularity Chip for Analysis of Rare Targets (SMART-Chip): Liquid Biopsy Samples.

Liquid biopsies are becoming popular for managing a variety of diseases due to the minimally invasive nature of their acquisition, thus potentially providing better outcomes for patients. Circulating tumor cells (CTCs) are among the many different biomarkers secured from a liquid biopsy, and a number of efficient platforms for their isolation and enrichment from blood have been reported. However, many of these platforms require manual sample handling, which can generate difficulties when translating CTC assays into the clinic due to potential sample loss, contamination, and the need for highly specialized operators. We report a system modularity chip for the analysis of rare targets (SMART-Chip) composed of three task-specific modules that can fully automate processing of CTCs. The modules were used for affinity selection of the CTCs from peripheral blood with subsequent photorelease, simultaneous counting, and viability determinations of the CTCs and staining/imaging of the CTCs for immunophenotyping. The modules were interconnected to a fluidic motherboard populated with valves, interconnects, pneumatic control channels, and a fluidic network. The SMART-Chip components were made from thermoplastics via microreplication, which lowers the cost of production making it amenable to clinical implementation. The utility of the SMART-Chip was demonstrated by processing blood samples secured from colorectal cancer (CRC) and pancreatic ductal adenocarcinoma (PDAC) patients. We were able to affinity-select EpCAM expressing CTCs with high purity (0-3 white blood cells/mL of blood), enumerate the selected cells, determine their viability, and immunophenotype the cells. The assay could be completed in <4 h, while manual processing required >8 h.

Investigating Customization Strategies and Convergence Behaviors of Task-Specific ADMM.

Alternating Direction Method of Multiplier (ADMM) has been a popular algorithmic framework for separable optimization problems with linear constraints. For numerical ADMM fail to exploit the particular structure of the problem at hand nor the input data information, leveraging task-specific modules (e.g., neural networks and other data-driven architectures) to extend ADMM is a significant but challenging task. This work focuses on designing a flexible algorithmic framework to incorporate various task-specific modules (with no additional constraints) to improve the performance of ADMM in real-world applications. Specifically, we propose Guidance from Optimality (GO), a new customization strategy, to embed task-specific modules into ADMM (GO-ADMM). By introducing an optimality-based criterion to guide the propagation, GO-ADMM establishes an updating scheme agnostic to the choice of additional modules. The existing task-specific methods just plug their task-specific modules into the numerical iterations in a straightforward manner. Even with some restrictive constraints on the plug-in modules, they can only obtain some relatively weaker convergence properties for the resulted ADMM iterations. Fortunately, without any restrictions on the embedded modules, we prove the convergence of GO-ADMM regarding objective values and constraint violations, and derive the worst-case convergence rate measured by iteration complexity. Extensive experiments are conducted to verify the theoretical results and demonstrate the efficiency of GO-ADMM.

Joint Super-Resolution and Alignment of Tiny Faces

Super-resolution (SR) and landmark localization of tiny faces are highly correlated tasks. On the one hand, landmark localization could obtain higher accuracy with faces of high-resolution (HR). On the other hand, face SR would benefit from prior knowledge of facial attributes such as landmarks. Thus, we propose a joint alignment and SR network to simultaneously detect facial landmarks and super-resolve tiny faces. More specifically, a shared deep encoder is applied to extract features for both tasks by leveraging complementary information. To exploit representative power of the hierarchical encoder, intermediate layers of a shared feature extraction module are fused to form efficient feature representations. The fused features are then fed to task-specific modules to detect landmarks and super-resolve face images in parallel. Extensive experiments demonstrate that the proposed model significantly outperforms the state-of-the-art in both landmark localization and SR of faces. We show a large improvement for landmark localization of tiny faces (i.e., 16 × 16). Furthermore, the proposed framework yields comparable results for landmark localization on low-resolution (LR) faces (i.e., 64 × 64) to existing methods on HR (i.e., 256 × 256). As for SR, the proposed method recovers sharper edges and more details from LR face images than other state-of-the-art methods, which we demonstrate qualitatively and quantitatively.

Modularity underlying the performance of unusual locomotor tasks inspired by developmental milestones.

Motor behaviors are often hypothesized to be set up from the combination of a small number of modules encoded in the central nervous system. These modules are thought to combine such that a variety of motor tasks can be realized, from reproducible tasks such as walking to more unusual locomotor tasks that typically exhibit more step-by-step variability. We investigated the impact of step-by-step variability on the modular architecture of unusual tasks compared with walking. To this aim, 20 adults had to perform walking and two unusual modes of locomotion inspired by developmental milestones (cruising and crawling). Sixteen surface electromyography (EMG) signals were recorded to extract both spatial and temporal modules. Modules were extracted from both averaged and nonaveraged (i.e., single step) EMG signals to assess the significance of step-to-step variability when participants practiced such unusual locomotor tasks. The number of modules extracted from averaged data was similar across tasks, but a higher number of modules was required to reconstruct nonaveraged EMG data of the unusual tasks. Although certain walking modules were shared with cruising and crawling, task-specific modules were necessary to account for the muscle patterns underlying these unusual locomotion modes. These results highlight a more complex modularity (e.g., more modules) for cruising and crawling compared with walking, which was only apparent when the step-to-step variability of EMG patterns was considered. This suggests that considering nonaveraged data is relevant when muscle modularity is studied, especially in motor tasks with high variability as in motor development.NEW & NOTEWORTHY This study addresses the general question of modularity in locomotor control. We demonstrate for the first time the importance of intraindividual variability in the muscle modularity of unusual locomotor behaviors that exhibit greater step-by-step variability than standard walking. Crawling and cruising, the unusual locomotor modes considered, are based on a more complex modular organization than walking. More spatial and temporal modules, task specific or shared with walking modules, are needed to reconstruct muscle patterns.

Region-Based Context Enhanced Network for Robust Multiple Face Alignment

The recent studies for face alignment have involved developing an isolated algorithm on well-cropped face images. It is difficult to obtain the expected input by using an off-the-shelf face detector in practical applications. In this paper, we attempt to bridge between face detection and face alignment by establishing a novel joint multi-task model, which allows us to simultaneously detect multiple faces and their landmarks on a given scene image. In contrast to the pipeline-based framework by cascading separate models, we aim to propose an end-to-end convolutional network by sharing and transform feature representations between the task-specific modules. To learn a robust landmark estimator for unconstrained face alignment, three types of context enhanced blocks are designed to encode feature maps with multi-level context, multi-scale context, and global context. In the post-processing step, we develop a shape reconstruction algorithm based on point distribution model to refine the landmark outliers. Extensive experiments demonstrate that our results are robust for the landmark location task and insensitive to the location of estimated face regions. Furthermore, our method significantly outperforms recent state-of-the-art methods on several challenging datasets including 300 W, AFLW, and COFW.

Denoising with kernel prediction and asymmetric loss functions

We present a modular convolutional architecture for denoising rendered images. We expand on the capabilities of kernel-predicting networks by combining them with a number of task-specific modules, and optimizing the assembly using an asymmetric loss. The source-aware encoder---the first module in the assembly---extracts low-level features and embeds them into a common feature space, enabling quick adaptation of a trained network to novel data. The spatial and temporal modules extract abstract, high-level features for kernel-based reconstruction, which is performed at three different spatial scales to reduce low-frequency artifacts. The complete network is trained using a class of asymmetric loss functions that are designed to preserve details and provide the user with a direct control over the variance-bias trade-off during inference. We also propose an error-predicting module for inferring reconstruction error maps that can be used to drive adaptive sampling. Finally, we present a theoretical analysis of convergence rates of kernel-predicting architectures, shedding light on why kernel prediction performs better than synthesizing the colors directly, complementing the empirical evidence presented in this and previous works. We demonstrate that our networks attain results that compare favorably to state-of-the-art methods in terms of detail preservation, low-frequency noise removal, and temporal stability on a variety of production and academic datasets.

KALIS - An eHealth System for Biomedical Risk Analysis of Drugs.

In Germany, adverse drug reactions and events cause hospitalizations, which lead to numerous thousands of deaths and several million Euros in additional health costs annually. Approximately one in two deaths could be avoided by an appropriate system for risk analysis of drugs. The integration and storage of several data sources from life sciences are an ongoing need to address various questions with respect to drug therapy. A software architecture for data integration was implemented in order to build up a new data warehouse named KALIS-DWH, which includes pharmacological, biomolecular and patient-related data. Based on this comprehensive KALIS-DWH, an eHealth system named KALIS for biomedical risk analysis of drugs was implemented. The task-specific modules of KALIS offer efficient algorithms for analyzing medication and supporting decision-making in drug therapy. KALIS is meant to be a web-based information system for health professionals and researchers. KALIS provides comprehensive knowledge and modules for risk analysis of drugs, which can contribute to minimizing prescribing errors.

Modular microfluidic system fabricated in thermoplastics for the strain-specific detection of bacterial pathogens.

The recent outbreaks of a lethal E. coli strain in Germany have aroused renewed interest in developing rapid, specific and accurate systems for detecting and characterizing bacterial pathogens in suspected contaminated food and/or water supplies. To address this need, we have designed, fabricated and tested an integrated modular-based microfluidic system and the accompanying assay for the strain-specific identification of bacterial pathogens. The system can carry out the entire molecular processing pipeline in a single disposable fluidic cartridge and detect single nucleotide variations in selected genes to allow for the identification of the bacterial species, even its strain with high specificity. The unique aspect of this fluidic cartridge is its modular format with task-specific modules interconnected to a fluidic motherboard to permit the selection of the target material. In addition, to minimize the amount of finishing steps for assembling the fluidic cartridge, many of the functional components were produced during the polymer molding step used to create the fluidic network. The operation of the cartridge was provided by electronic, mechanical, optical and hydraulic controls located off-chip and packaged into a small footprint instrument (1 ft(3)). The fluidic cartridge was capable of performing cell enrichment, cell lysis, solid-phase extraction (SPE) of genomic DNA, continuous flow (CF) PCR, CF ligase detection reaction (LDR) and universal DNA array readout. The cartridge was comprised of modules situated on a fluidic motherboard; the motherboard was made from polycarbonate, PC, and used for cell lysis, SPE, CF PCR and CF LDR. The modules were task-specific units and performed universal zip-code array readout or affinity enrichment of the target cells with both made from poly(methylmethacrylate), PMMA. Two genes, uidA and sipB/C, were used to discriminate between E. coli and Salmonella, and evaluated as a model system. Results showed that the fluidic system could successfully identify bacteria in <40 min with minimal operator intervention and perform strain identification, even from a mixed population with the target of a minority. We further demonstrated the ability to analyze the E. coli O157:H7 strain from a waste-water sample using enrichment followed by genotyping.

Evolvable Production Systems: Environment for New Business Models

The increasing threats and opportunities, on the global markets, challenge today the company at several levels. The actual Manufacturing Systems becomes dramatic for the survival and prosperity of the organization. Agility, Sustainability and high rate of Re-usability are the main objectives of an “Evolvable Production System”. Task-specific and process-oriented modules with embedded intelligence, together with distributed control based on agent technology are the cornerstones of an EPS. The realization and management of this kind of systems entail an ontological approach due to their implicit complexity. The reference architecture is the key to canalize the underlying holism of an EPS towards an effective instantiation, which exploits the “Emergent Behavior”, the key driver of the Evolvability of a system. The issues related with the application of this view, such as dynamic link between product and system design as well as the management of a module repository, underline the weakness of present business models. Therefore a model for the emplacement of a System that follows the “Evolvable Paradigm” predicament has been produced in order to enable further studies. One of the conclusions of the work is the need for a more suitable approach to business by the main actors involved in the realization of an EPS. Sharing information and standards about product design and modules realization becomes a compulsory strategic choice.

Content- and Task-Specific Dissociations of Frontal Activity during Maintenance and Manipulation in Visual Working Memory

Working memory, the short-term maintenance and manipulation of information, relies strongly on neural activity in the frontal cortex. Understanding the functional role of this activity is a prerequisite for the understanding of cognitive control mechanisms. Functional imaging studies in human participants have attempted to reveal neural correlates of the subdivision of visual working memory into different processes (maintenance vs manipulation) and according to the type of memorized content. Here, we show, using functional magnetic resonance imaging, a content-specific dissociation of frontal activity, with dorsal premotor areas supporting both maintenance and manipulation of spatial features and more ventral areas supporting maintenance and manipulation of color. Manipulation-specific activity was observed in the anterior middle frontal gyrus, the inferior frontal junction, and the inferior parietal lobe bilaterally. These areas have been implicated in cognitive control, and their activation by the manipulation task conforms to the demand on central executive resources in this condition. We suggest that the enhanced demand on cognitive resources in manipulation compared with maintenance was met by interplay of content- and task-specific modules in a frontoparietal network.

Functional proteomic analysis of a three-tier PKCε-Akt-eNOS signaling module in cardiac protection

Cardiac protective signaling networks have been shown to involve PKCepsilon. However, the molecular mechanisms by which PKCepsilon interacts with other members of these networks to form task-specific modules remain unknown. Among 93 different PKCepsilon-associated proteins that have been identified, Akt and endothelial nitric oxide (NO) synthase (eNOS) are of importance because of their independent abilities to promote cell survival and prevent cell death. The simultaneous association of PKCepsilon, Akt, and eNOS has not been examined, and, in particular, the formation of a module containing these three proteins and the role of such a module in the regulation of NO production and cardiac protection are unknown. The present study was undertaken to determine whether these molecules form a signaling module and, thereby, play a collective role in cardiac signaling. Using recombinant proteins in vitro and PKCepsilon transgenic mouse hearts, we demonstrate the following: 1) PKCepsilon, Akt, and eNOS interact and form signaling modules in vitro and in the mouse heart. Activation of either PKCepsilon or Akt enhances the formation of PKCepsilon-Akt-eNOS signaling modules. 2) PKCepsilon directly phosphorylates and enhances activation of Akt in vitro, and PKCepsilon activation increases phosphorylation and activation of Akt in PKCepsilon transgenic mouse hearts. 3) PKCepsilon directly phosphorylates eNOS in vitro, and this phosphorylation enhances eNOS activity. Activation of PKCepsilon in vivo increased phosphorylation of eNOS at Ser(1177), indicating eNOS activation. This study characterizes, for the first time, the physical, as well as functional, coupling of PKCepsilon, Akt, and eNOS in the heart and implicates these PKCepsilon-Akt-eNOS signaling modules as critical signaling elements during PKCepsilon-induced cardiac protection.

Learning to Optimize Performance: Lessons from a Neural Control System

Cognitive processes in human-machine interaction are mostly executed in the cerebral cortex (and other parts of the higher brain) which is designed for multi-tasking operations but is generally not optimized for any specific task. Similar cognitive and control operations also occur in the brain stem which is composed of many task specific neural modules for the homeostatic regulation of a host of physiological functions using peripheral processes as the “machine”. In the respiratory control system, the brain stem controller demonstrates intelligent control capability and achieves optimal performance under a wide variety of environmental conditions. Such intelligent control behavior can be realized by reinforcement learning in an associative Hebbian synapse in the brain stem. Similar reinforcement learning modules in the higher brain can be fashioned by repeated training to effect robust optimal operation in a human-machine system.