Deep Module Research Articles

Data- and Model-Driven Crude Oil Supply Risk Assessment of China Considering Maritime Transportation Factors

Effective supply-chain risk assessment is the basis for developing sustainable supply policies, and it has received growing attention in global oil supply system management. Dynamical modeling and data-driven modeling are two main risk assessment technologies that have been applied in crude oil supply networks. Dynamical risk modeling and data-driven risk modeling offer distinct advantages in capturing the complexities and dynamics of the system. Considering their complementary strengths, a hybrid modeling framework combining system dynamics and data-driven neural networks is proposed for risk assessment of crude oil transportation network. Specifically, the system dynamics module is to capture and interpret the underlying dynamics and mechanisms of the transportation network, while the deep neural networks module is to discover the nonlinear patterns and dependencies of risk factors from various inputs. Based on joint training, the hybrid model can ultimately develop the capability of risk prediction with a small amount of data. In addition, it can consider the dynamic nature of crude oil transportation networks to interpret the predicted results of the risk level for decision-makers to make specific risk-mitigating policies. Extensive experiments based on China’s scenario have been conducted to demonstrate the effectiveness of the proposed hybrid model, and the results show that our model achieves higher accuracy in risk prediction compared to the current state of the art. The results also present an explanation for China’s policy change of building a resilient crude oil transportation system.

A component-based design approach for energy flexibility in cyber-physical manufacturing systems

AbstractEnergy flexibility of manufacturing systems helps to meet sustainable manufacturing requirements and is getting significant importance with rising energy prices and noticeable climate changes. Considering the need to proactively enable energy flexibility in modern manufacturing systems, this work presents a component-based design approach that aims to embed energy flexibility in the design of cyber-physical production systems. To this end, energy management using Industry 4.0 technologies (e.g., Internet of Things and Cyber-physical Systems) is compared to the literature on energy flexibility in order to evaluate to what extent the energy flexibility practice takes advantage of Industry 4.0 technologies. Another dimension is the coverage of the life cycle of the manufacturing system which guarantees its sustainable design and the ability to achieve energy flexibility by configuring the energy consumption behaviour. A data-based design approach of the energy-flexible components is proposed in the spirit of the Reference Architectural Model Industrie 4.0 (RAMI 4.0), and then it is exemplified using an electric drive (as a component) in order to show the practical applicability of the approach. The energy consumption behaviour of the component is modelled using machine learning techniques. The digital twin of the studied component is developed using Visual Components virtual engineering environment, then its energy consumption behaviour is included in the model allowing the system integrator/decision-maker to configure the component based on the energy availability/price. Finally, external services in terms of an optimisation module and a deep learning module are connected to the digital twin.

A encoder-decoder deblurring network combined with high-frequency a priori

Convolutional neural network based methods have been proposed to address blind deblurring. Aiming at the difficulty of reconstructing high-frequency features of images with existing network models, this paper proposes a two-stage convolution-based encoder-decoder for fusing high-frequency a prior. In the first stage, both blurred and high-frequency images are input, and the image features extracted by the network and the high-frequency features are fused. In the second stage, the fused features are further refined and recovered as potentially clear images, and the reconstruction ability of the model for high-frequency features is enhanced. In addition, this paper proposes a deep feature reorganization module that integrates multi-layer semantic information in the encoder-decoder and targets the encoder semantics to further enhance the feature characterization capability of the model. Comprehensive experimental results show that our method achieves 0.9085 structural similarity index (SSIM) and 30.66db peak signal-to-noise ratio (PSNR) on the GoPro dataset. Meanwhile, our method achieves 0.8514 SSIM and 27.39db PSNR on the Lai dataset.

RepECN: Making ConvNets Better Again for Efficient Image Super-Resolution

RepECN: Making ConvNets Better Again for Efficient Image Super-Resolution

Chinese Fine-Grained Named Entity Recognition Based on BILTAR and GlobalPointer Modules

The task of fine-grained named entity recognition is to locate entities in text and classify them into predefined fine-grained categories. At present, Chinese fine-grained NER only uses the pretrained language model to encode the characters in the sentence and lacks the ability to extract the deep semantic, sequence, and position information. The sequence annotation method is character-based and lacks the processing of entity boundaries. Fine-grained entity categories have a high degree of similarity, which makes it difficult to distinguish similar categories. To solve the above problems, this paper constructs the BILTAR deep semantic extraction module and adds the GlobalPointer module to improve the accuracy of Chinese fine-grained named entity recognition. The BILTAR module is used to extract deep semantic features from the coding information of pretrained language models and use higher-quality features to improve the model performance. In the GlobalPointer module, the model first adds the rotation position encoding information to the feature vector, using the position information to achieve data enhancement. Finally, the model considers all possible entity boundaries through the GlobalPointer module and calculates the scores for all possible entity boundaries in each category. In this paper, all possible entity boundaries in the text are considered by the above method, and the accuracy of entity recognition is improved. In this paper, the corresponding experiments were carried out on CLUENER 2020 and the micro Chinese fine-grained NER dataset, and the F1 scores of the model in this paper reached 80.848% and 75.751%, respectively. In ablation experiments, the proposed method outperforms the most advanced baseline model and improves the performance of the basic model.

Container port truck dispatching optimization using Real2Sim based deep reinforcement learning

In marine container terminals, truck dispatching optimization is often considered as the primary focus as it provides crucial synergy between the sea-side operations and yard-side activities and hence can greatly affect the terminal throughput and quay crane utilization. However, many existing studies rely on strong assumptions that often overlook the uncertainties and dynamics innate to real-life applications. In this work, we propose a dynamic truck dispatching system for container ports equipped with the latest IoT technologies. The system is comprised of Real2Sim simulation and a truck dispatch agent, trained through a spatial-attention based deep reinforcement learning module, supported by an expert network. The proposed Real2Sim framework has the ability to model the non-linear complexities and non-deterministic events while our attention-aware deep reinforcement learning module is capable of making full use of both historical and real-time port data to learn a high-quality truck dispatching policy under uncertainties. Extensive experiments show our proposed method has good generalization and achieves the state-of-the-art results on the problems derived from real-life data of a large international port.

A Pre-Procession Module for Point-Based Deep Learning in Dense Point Clouds in the Ship Engineering Field

Recently, point cloud technology has been applied in the ship engineering field. However, the dense point cloud acquired by terrestrial laser scanning (TLS) technology in ship engineering applications brings an obstacle to some powerful and advanced point-based deep learning point cloud processing methods. This paper presents a deep learning pre-procession module to ensure the feasibility of processing dense point clouds on commonly available computer devices. The pre-procession module is designed according to the traditional point cloud processing methods and the PointNet++ paradigm, and is evaluated on two ship structure datasets and two popular point cloud datasets. Experimental results illustrate that (i) the proposed module improves the performance of point-based deep learning semantic segmentation networks, and (ii) the proposed module empowers the existing point-based deep learning networks with the capability to process dense input point clouds. The proposed module may provide a useful semantic segmentation tool for realistic dense point clouds in various industrial applications.

A filter-augmented auto-encoder with learnable normalization for robust multivariate time series anomaly detection

While existing reconstruction-based multivariate time series (MTS) anomaly detection methods demonstrate advanced performance on many challenging real-world datasets, they generally assume the data only consists of normal samples when training models. However, real-world MTS data may contain significant noise and even be contaminated by anomalies. As a result, most existing approaches easily capture the pattern of the contaminated data, making identifying anomalies more difficult. Although a few studies have aimed to mitigate the interference of the noise and anomalies by introducing various regularizations, they still employ the objective of fully reconstructing the input data, impeding the model from learning an accurate profile of the MTS’s normal pattern. Moreover, it is difficult for existing methods to apply the most appropriate normalization schemes for each dataset in various complex scenarios, particularly for mixed-feature MTS. This paper proposes a filter-augmented auto-encoder with learnable normalization (NormFAAE) for robust MTS anomaly detection. Firstly, NormFAAE designs a deep hybrid normalization module. It is trained with the backbone end-to-end in the current training task to perform the optimal normalization scheme. Meanwhile, it integrates two learnable normalization sub-modules to deal with the mixed-feature MTS effectively. Secondly, NormFAAE proposes a filter-augmented auto-encoder with a dual-phase task. It separates the noise and anomalies from the input data by a deep filter module, which facilitates the model to only reconstruct the normal data, achieving a more robust latent representation of MTS. Experimental results demonstrate that NormFAAE outperforms 17 typical baselines on five real-world industrial datasets from diverse fields.

DeepHLAPred: a deep learning-based method for non-classical HLA binder prediction

Human leukocyte antigen (HLA) is closely involved in regulating the human immune system. Despite great advance in detecting classical HLA Class I binders, there are few methods or toolkits for recognizing non-classical HLA Class I binders. To fill in this gap, we have developed a deep learning-based tool called DeepHLAPred. The DeepHLAPred used electron-ion interaction pseudo potential, integer numerical mapping and accumulated amino acid frequency as initial representation of non-classical HLA binder sequence. The deep learning module was used to further refine high-level representations. The deep learning module comprised two parallel convolutional neural networks, each followed by maximum pooling layer, dropout layer, and bi-directional long short-term memory network. The experimental results showed that the DeepHLAPred reached the state-of-the-art performanceson the cross-validation test and the independent test. The extensive test demonstrated the rationality of the DeepHLAPred. We further analyzed sequence pattern of non-classical HLA class I binders by information entropy. The information entropy of non-classical HLA binder sequence implied sequence pattern to a certain extent. In addition, we have developed a user-friendly webserver for convenient use, which is available at http://www.biolscience.cn/DeepHLApred/. The tool and the analysis is helpful to detect non-classical HLA Class I binder. The source code and data is available at https://github.com/tangxingyu0/DeepHLApred.

Boundary guided network with two-stage transfer learning for gastrointestinal polyps segmentation

The automated segmentation of polyps plays a crucial role in the early diagnosis and treatment of gastrointestinal diseases. However, due to the diversity of polyp lesions and complex imaging environment, the accurate identification of the true lesion area is challenging, especially for small polyps. The blurred boundary of polyps can also result in over or under-segmentation issues. This research proposes a boundary-guided network with two-stage transfer learning: (1) the network is trained to determine the region of interest for polyp lesions and save the initial weights; (2) transfer learning is applied to leverage the learned prior knowledge to perform fine segmentation of the region of interest. It can accurately identify the lesion area, thereby achieving good segmentations, especially for small polyps. Besides, the pyramid vision transformer is used as the feature backbone. Boundary feature extraction module (BFE), deep feature extraction module (DFE), and multi-scale fusion module (MF) are designed to generate boundary maps that guide the decoder in generating prediction maps. Experimental results show that the proposed method outperforms the comparative methods on four public datasets and a private dataset (including gastric polyps), with mDSC scores exceeding 85%. Notably, on the ETIS-Larib dataset, the mDSC score is improved by 11.7% compared to methods used for comparison.

Automatic Tracking Method for 3D Human Motion Pose Using Contrastive Learning

Automatic tracking of three-dimensional (3D) human motion pose has the potential to provide corresponding technical support in various fields. However, existing methods for tracking human motion pose suffer from significant errors, long tracking times and suboptimal tracking results. To address these issues, an automatic tracking method for 3D human motion pose using contrastive learning is proposed. By using the feature parameters of 3D human motion poses, threshold variation parameters of 3D human motion poses are computed. The golden section is introduced to transform the threshold variation parameters and extract the features of 3D human motion poses by comparing the feature parameters with the threshold of parameter variation. Under the supervision of contrastive learning, a constraint loss is added to the local–global deep supervision module of contrastive learning to extract local parameters of 3D human motion poses, combined with their local features. After normalizing the 3D human motion pose images, frame differences of the background image are calculated. By constructing an automatic tracking model for 3D human motion poses, automatic tracking of 3D human motion poses is achieved. Experimental results demonstrate that the highest tracking lag is 9%, there is no deviation in node tracking, the pixel contrast is maintained above 90% and only 6 sub-blocks have detail loss. This indicates that the proposed method effectively tracks 3D human motion poses, tracks all the nodes, achieves high accuracy in automatic tracking and produces good tracking results.

VHXLA: A post-earthquake damage prediction method for high-speed railway track-bridge system using VMD and hybrid neural network

The timely and accurate prediction of post-earthquake damage is critically important for ensuring the safety of high-speed railway track-bridge systems. The study introduces a novel method known as the VHXLA model. This innovative model blends variational mode decomposition (VMD) with a hybrid neural network for predicting non-linear multi-component post-earthquake damage in high-speed railways. The VHXLA model comprises a signal processing module and a deep learning module. The signal processing module utilises VMD decomposition, and the Hilbert transform (HT) to transform two-dimensional seismic signals into four-dimensional complex time–frequency signals, thus simplifying the task of identifying seismic time–frequency characteristics via the neural network. The deep learning module integrates diverse types of neural network components, such as the Xception CNN feature extraction submodule, the LSTM RNN temporal learning submodule, and the multi-head attention mechanism submodule. A Bayesian self-optimisation method is implemented to determine the number of decomposition layers in VMD and select essential hyperparameters. The model's effectiveness is evaluated by predicting the damage of an experimentally validated finite element model, subjecting it to seismic loads, and thus gauging its performance. The results indicated that the signal processing module, based on VMD decomposition, significantly improves the neural network's signal processing capability. In addition, the synergistic integration of different modules in the VHXLA model provides superior prediction accuracy compared to pre-existing damage prediction models. Notably, the prediction accuracy is consistent across different positions of the same predicted component in the high-speed railway track-bridge system.

MWDINet: A multilevel wavelet decomposition interaction network for stock price prediction

Stock price prediction is a classical interdisciplinary issue drawn from finance, computer science, econometrics, and mathematics. Most stock price data are nonlinear, nonstationary, and highly complex, making stock price prediction challenging. Recently, deep neural networks (DNNs) have demonstrated powerful learning capabilities and have yielded notable results in stock price prediction tasks. Most existing deep learning solutions, however, only consider time-domain information or lack effective modeling of frequency-domain information, thus failing to effectively utilize both time-domain and frequency-domain information of the data. Meanwhile, existing methods ignore autocorrelated errors in the stock price forecasting task due to missing valid information data, i.e., they do not consider the correlation between the error at the current time step and the error at the previous time step, which undermines the standard maximum likelihood estimation (MLE) assumption, thereby weakening the model’s performance. We propose a multilevel wavelet decomposition interaction network (MWDINet), an end-to-end framework for stock price prediction. MWDINet employs the multiscale wavelet decomposition interaction module (MWDI-Block) and the Hull Moving Average module (HMA-Block) to extract the data’s frequency-domain and time-domain information, respectively. In MWDI-Block, the traditional signal processing method of Maximum Overlapping Discrete Wavelet Transform (MODWT) is seamlessly embedded into a deep learning framework (named DMODWT). The DMODWT algorithm not only automatically extracts the frequency-domain information from the data, but also fine-tunes the wavelet filters. With HMA-Block, we improve the Hull Moving Average (HMA), commonly used in the industry, into a deep learning module, which learns how changes in different markets over time. Inspired by the research on correcting autocorrelated errors in linear models in econometrics, we further design a deep difference module (DIF-Block) to correct autocorrelated errors and thus improve the prediction performance of the model. Moreover, all components are integrated seamlessly in a unified end-to-end framework. Extensive experiments on real-world datasets demonstrate that MWDINet outperforms the state-of-the-art models and has remarkable potential in stock price prediction.

DSCA-Net: Double-stage Codec Attention Network for automatic nuclear segmentation

The rapid and precise segmentation of cell nuclei from hematoxylin and eosin-stained tissue images is an essential clinical undertaking with significant implications for various clinical applications. The segmentation of cell nuclei poses specific challenges due to the inherent instability of nuclear morphology and the complexity of the segmentation environments. Furthermore, previous studies have primarily relied on small-scale and limited-diverse datasets, potentially hindering their applicability to clinical tasks. This study introduces a novel approach, the Double-stage Codec Attention Network, designed to automatically and accurately segment nuclei. Specifically, we present a hierarchical feature extraction module, which maximizes the utilization of cell nuclei’s morphological characteristics in the tissue, thereby providing critical semantic information for nucleus segmentation. Furthermore, the feature selection units are employed to enhance relevant features and suppress interfering ones, thereby enhancing the overall expressive capacity of the information. The multi-scale deep feature fusion module utilizes interrelated encoder–decoder connections to jointly optimize and integrate this information, generating a robust hierarchical feature pyramid. Finally, the feature attention fusion mechanism captures spatial and directional information, aiding the model in the accurate localization and recognition of cell nuclei. We rigorously evaluated our proposed method using the PanNuke dataset, the largest comprehensive histology dataset of cancer tissues. In terms of the average F1-score across all segmentation classes in the PanNuke dataset, DSCA-Net outperforms other state-of-the-art models such as DeepLabV3+, TransUNet, Triple U-net, and TransNuSeg by 1.38, 1.44, 2.64, and 1.02, respectively. Additionally, DSCA-Net shows excellent efficiency in generating predictive images, outperforming all comparative models.

Back projection deep unrolling network for handwritten text image super resolution

Current super-resolution (SR) methods have demonstrated exceptional advancements in the domain of natural image processing. Nevertheless, these approaches do not fully address the open issues of edge blurring and distortion. In order to address the aforementioned challenges, we propose a novel deep unrolling model, coined back projection deep unrolling network (BPDUN), for super-resolution of handwritten text images leveraging the Algorithm Unrolling paradigm. We design the network model of BPDUN by unfolding and truncating the traditional iterative back projection algorithm (IBPA). We unroll IBPA into a cascade operation of three building blocks (deep denoising module, low-frequency reconstruction module, and residual projection module). BPDUN inherits the interpretability of the iterative optimization algorithm and is also designed to make the reconstructed text image more realistic and natural. Moreover, we propose a new benchmark dataset to address challenging SR problems of handwritten text image (HDT300). Extensive experiments show that BPDUN obtains an enhanced balance between the performance (quantified by PSNR and SSIM) and the cost (as measured by network parameters). Notably, BPDUN sets new benchmarks on the HDT300 dataset, surpassing previous state-of-the-art approaches by achieving up to 0.2 dB gains in PSNR.

Comprehensive urban space representation with varying numbers of street-level images

Street-level imagery has emerged as a valuable tool for observing large-scale urban spaces with unprecedented detail. However, previous studies have been limited to analyzing individual street-level images. This approach falls short in representing the characteristics of a spatial unit, such as a street or grid, which may contain varying numbers of street-level images ranging from several to hundreds. As a result, a more comprehensive and representative approach is required to capture the complexity and diversity of urban environments at different spatial scales. To address this issue, this study proposes a deep learning-based module called Vision-LSTM, which can effectively obtain vector representation from varying numbers of street-level images in spatial units. The effectiveness of the module is validated through experiments to recognize urban villages, achieving reliable recognition results (overall accuracy: 91.6%) through multimodal learning that combines street-level imagery with remote sensing imagery and social sensing data. Compared to existing image fusion methods, Vision-LSTM demonstrates significant effectiveness in capturing associations between street-level images. The proposed module can provide a more comprehensive understanding of urban spaces, enhancing the research value of street-level imagery and facilitating multimodal learning-based urban research. Our models are available at https://github.com/yingjinghuang/Vision-LSTM.

Application of aesthetics in architectural design under the background of big data and artificial intelligence

Architectural aesthetics improve the appearance and value of a building/construction structure based on shape, color, rigidity, etc., appealingly. It includes the maximum safety requirements, durability, structural ability, etc. Therefore the aesthetic implementation requires high-level data accumulation and analysis to satisfy the earlier constraints. This article develops a Selective Aesthetic Application Paradigm (SAAP) for meeting the user criteria in structural design for region-specific adaptability. The proposed paradigm gathers information on the region, people’s expectations, visibility, and structural performance for the aesthetic design application. The proportion considerations in the application are subject to vary according to the region’s adaptability and performance. The proportion of the accumulated data influence in the application is determined using deep learning. In the learning paradigm, two-layered configurations for region-adaptability and performance measures are trained to provide aesthetic design application recommendations. Based on the suggestion and recommendation, the deep learning module is trained to rectify design errors. The training is independent of the previous two error and adaptability verification layers. It is performed using the qualified (selected) aesthetic design with a previous history of user satisfaction.

A novel wavelet-transform-based convolution classification network for cervical lymph node metastasis of papillary thyroid carcinoma in ultrasound images.

Preoperative assessment of cervical lymph nodes metastasis (CLNM) for accurate qualitative and locating diagnosis is important for choosing the best treatment option for patients with papillary thyroid cancer. Non-destructive, non-invasive ultrasound is currently the imaging method of choice for lymph node metastatic assessment. For lymph node characteristics and ultrasound images, this paper proposes a multitasking network framework for diagnosing metastatic lymph nodes in ultrasound images, in which localization module not only provides information on the location of lymph nodes to focus on the peripheral and self regions of lymph nodes, but also provides structural features of lymph nodes for subsequent classification module. In the classification module, we design a novel wavelet-transform-based convolution network. Wavelet transform is introduced into the deep learning convolution module to analyze ultrasound images in both spatial and frequency domains, which effectively enriches the feature information and improves the classification performance of the model without increasing the model parameters. We collected 510 patient data (N = 1376) from Shanghai Sixth People's Hospital regarding ultrasound lymph nodes in the neck, as well as used three publicly available ultrasound datasets, including SCUI2020 (N = 2914), DDTI (N = 480), and BUSI (N = 780). Compared to the optimal two-stage model, our model has improved its accuracy and AUC indexes by 5.83% and 4%, which outperforms the two-stage architectures and also surpasses the latest classification networks.

KaryoNet: Chromosome Recognition with End-to-End Combinatorial Optimization Network.

Chromosome recognition is a critical way to diagnose various hematological malignancies and genetic diseases, which is however a repetitive and time-consuming process in karyotyping. To explore the relative relation between chromosomes, in this work, we start from a global perspective and learn the contextual interactions and class distribution features between chromosomes within a karyotype. We propose an end-to-end differentiable combinatorial optimization method, KaryoNet, which captures long-range interactions between chromosomes with the proposed Masked Feature Interaction Module (MFIM) and conducts label assignment in a flexible and differentiable way with Deep Assignment Module (DAM). Specially, a Feature Matching Sub-Network is built to predict the mask array for attention computation in MFIM. Lastly, Type and Polarity Prediction Head can predict chromosome type and polarity simultaneously. Extensive experiments on R-band and G-band two clinical datasets demonstrate the merits of the proposed method. For normal karyotypes, the proposed KaryoNet achieves the accuracy of 98.41% on R-band chromosome and 99.58% on G-band chromosome. Owing to the extracted internal relation and class distribution features, KaryoNet can also achieve state-of-the-art performances on karyotypes of patients with different types of numerical abnormalities. The proposed method has been applied to assist clinical karyotype diagnosis. Our code is available at: https://github.com/xiabc612/KaryoNet.

Robust semantic segmentation of retinal fluids from SD-OCT images using FAM-U-Net

Semantic segmentation of retinal fluids is a crucial step in detecting vision threatening abnormalities like Macular Edema (ME), Diabetic Retinopathy, etc. These fluids accumulate between the retinal layers resulting in the damage of photoreceptor cells and causing visual impairment to the patients. As a result, an early and accurate diagnosing system is essential to reduce the spread of these fluids over the retina. With the advancement of Optical coherence tomography (OCT) imaging and artificial intelligence techniques in the eye care sector, a novel deep learning network, FAM-U-Net (Multiscale Feature Aggregation and Double Attention Mixed U-Net) based on the U-Net architecture is implemented by combining Multiscale Feature Extraction (MFE) module, Deep Aggregation Pyramid Pooling Module (DAPPM) to extract the most pertinent features for fluid detection from OCT images. In addition to this, Convolution Block Attention Module (CBAM) is employed in the skip connection paths to enhance the regeneration and interpretation ability of the proposed network. Further, the performance of the proposed FAM-U-Net is evaluated using publicly available datasets like RETOUCH, OPTIMA, and DUKE. Consequently, the proposed model illustrates its superiority over other existing models by achieving an average Dice coefficient of 0.887, 0.786, and 0.821 for RETOUCH, OPTIMA, and DUKE datasets respectively.