Industrial Scene Research Articles

SCANet: Implementation of Selective Context Adaptation Network in Smart Farming Applications.

In the last decade, deep learning has enjoyed its spotlight as the game-changing addition to smart farming and precision agriculture. Such development has been predominantly observed in developed countries, while on the other hand, in developing countries most farmers especially ones with smallholder farms have not enjoyed such wide and deep adoption of this new technologies. In this paper we attempt to improve the image classification part of smart farming and precision agriculture. Agricultural commodities tend to possess certain textural details on their surfaces which we attempt to exploit. In this work, we propose a deep learning based approach called Selective Context Adaptation Network (SCANet). SCANet performs feature enhancement strategy by leveraging level-wise information and employing context selection mechanism. In exploiting contextual correlation feature of the crop images our proposed approach demonstrates the effectiveness of the context selection mechanism. Our proposed scheme achieves 88.72% accuracy and outperforms the existing approaches. Our model is evaluated on the cocoa bean dataset constructed from the real cocoa bean industry scene in Indonesia.

REG-Net: Improving 6DoF Object Pose Estimation With 2D Keypoint Long-Short-Range-Aware Registration

TheSixdegrees of freedom 6DoF pose estimation of texture-less objects provides a spatial understanding of industrial scenes and is the basis for accurate object manipulation. Recent studies have shown that the introduction of known model information and initial poses helps CNNs-based methods achieve better performance in complex scenes. However, the mapping from image space to pose space learned by neural networks is dimensionally lifting. Due to the lack of depth information, it is difficult for neural networks to capture local clues on texture-less surfaces and directly regress the relative 3-D translation and 3-D rotation. Instead, we propose a novel framework named REG-Net, which transforms the 6DoF pose estimation task into a 2-D keypoint registration problem. The proposed network first encodes regional prior information using multi-representation, utilizes the globally-consistent offset attention module to align 2-D keypoint features in a long range, and then estimates offsets and potential regions of keypoints. The proposed regional PnP simultaneously adjusts the keypoint locations in a short range and outputs the pose. This framework compresses the learning space of the network from 3-D to 2-D. Extensive experiments on two benchmark datasets demonstrate the robustness and accuracy of REG-Net. We further demonstrate the effectiveness of REG-Net in the reflective industrial part grasping applications.

Semantic Segmentation of Substation Site Cloud Based on Seg-PointNet

3D point cloud semantic segmentation has been widely used in industrial scenes and has attracted continuous attention as a critical technology for understanding the intelligent robot scene. However, extracting visual semantics in complex environments remains a challenge. We propose the Seg-PointNet model based on multi-layer residual structure and feature pyramid for the LiDAR point cloud data semantic segmentation task in the complex substations scene. The model is based on the PointNet network and introduces a multi-scale residual structure. The residual structure multilayer perception (RES-MLP) model is proposed to fully excavate features at different levels and improve the characterization capabilities of complex features. Moreover, the 3D point cloud feature pyramid module is proposed to characterize the substation scene’s semantic features. We tested and verified the Seg-PointNet model on a self-built substation cloud point (SCP) dataset. The results show that the proposed Seg-PointNet model effectively improves the point cloud data segmentation accuracy, with an accuracy of 89.23% and mean intersection over union (mIoU) of 63.57%. This shows that the model can be applied to substation scenarios and provide technical support to intelligent robots in complex substation environments.

Potential Requirements and Opportunities of Blockchain-Based Industrial IoT in Supply Chain: A Survey

The integration of the industrial Internet of Things (IoT) and blockchain technology is changing the business and management model of the supply chain. Much work is focused on how to promote the application of blockchain-based industrial IoT in the supply chain from both academic research and industrial practice. However, due to the different industrial requirements in industrial scenes, the gap between technology researches and industrial applications is still large. Therefore, the article adopts the mixed method of enterprise survey and literature review to identify the actual industrial requirements in different supply chain scenes. Also, the characteristics and applicable scenarios of industrial IoT and blockchain have been analyzed. Then, the potential application opportunities of blockchain-based industrial IoT in nine scenes are discussed in detail. This study reveals the technical challenges and practical challenges of these applications, which potentially guides research on applying industrial IoT and blockchain technology in the supply chain.

FIGAN: A Missing Industrial Data Imputation Method Customized for Soft Sensor Application

Missing data is quite common in the industrial field, resulting in problems in downstream applications, as most data driven methods used in these applications rely on complete and high-quality dataset to build a high-quality model. Existing methods deal with missing data individually regardless of its downstream application, treating all variables equally without considering their different roles in the downstream application. This would affect imputation performance for key variables, thus deteriorating the accuracy of the downstream model. A considerable challenge is how to refine the missing data imputation task. In this paper, a new method termed fine-tuned imputation GAN (FIGAN) is designed to achieve customized data imputation for industrial soft sensor. The major contribution of the paper lies in two aspects: 1) different from the original imputation GAN (GAIN) which treats all variables equally, FIGAN is guided by a soft sensor module so as to achieve customized data imputation by performing improved data imputation on quality-related variables. Enhanced accuracy for the final industrial soft sensor would be possible; 2) in addition, since labels of the soft sensor might also have missing data, a soft sensor with pseudo labeling is designed to conquer the problem with data imputation and label prediction being optimized interactively. Case studies on a converter steelmaking process and a penicillin fermentation process show the feasibility of the proposed FIGAN. It is noted that such customized imputation could be readily transferred to other downstream applications with missing data. Note to Practitioners—Industrial data is often incomplete and needs proper treatment. Meanwhile, downstream applications with preprocessed data vary under different industrial scenes. The focus of this study is to develop a customized data imputation method for specific downstream applications such as soft sensing. A fine-tuned imputation GAN is designed with a soft sensor module so as to guide the data imputation for good imputation on key variables of the soft sensor. Considering that missing data exists not only in measurement variables but also in soft sensor labels, a semi-supervised soft sensor is designed to handle missing data in the labels, optimized together with the imputation model. The customized data imputation can thus improve the final performance of the downstream model which is a soft sensor in this work. The customization could be transferred to other applications such as an anomaly detection model as well.

Transformer-Based Data-Driven Video Coding Acceleration for Industrial Applications

With the exploding development of edge intelligence and smart industry, deep learning-based intelligent industrial solutions are promptly applied in the manufacturing process. Many intelligent industrial solutions such as automatic manufacturing inspection are computer vision based and require fast and efficient video encoding techniques so that video streams can be processed as quickly as possible either at the edge cluster or over the cloud. As one of the most popular video coding standards, the high efficiency video coding (HEVC) standard has been applied to various industrial scenes. However, HEVC brings not only a higher compression rate but also a significant increase in encoding complexity, which hinders its practical application in industrial scenarios. Fortunately, a large amount of video coding data makes it possible to accelerate the encoding process in the industry. To speed up the video coding process in some industrial scenes, this paper proposes a data-driven fast approach for coding tree unit (CTU) partitioning in HEVC intracoding. First, we propose a method to represent the partition result of a CTU as a column vector of length 21. Then, we employ lots of encoding data produced in normal industry scenes to train transformer models used to predict the partitioning vector of the CTU. Finally, the final partitioning structure of the CTU is generated from the partitioning vector after a postprocessing operation and used by an industrial encoder. Compared with the original HEVC encoder used by some industrial applications, experiment results show that our approach achieves 58.77% encoding time reduction with 3.9% bit rate loss, which indicates that our data-driven approach for video coding has great capacity working in industrial applications.

Visually Equivalent Light Field 3-D for Portable Displays

Highly realistic 3-D displays that can reproduce object images to look like physical objects are utilized for natural and correct remote operation in industrial scenes. Therefore, we developed a visually equivalent light field 3-D (VELF3D) display that can produce highly realistic, accurate images with a high resolution and a smooth, accurate motion parallax. However, the observation distance is slightly long, and users cannot reach the displayed images. Therefore, we aim to develop a tablet-computer-type VELF3D display that enables users to touch the displayed objects. The display viewpoint density has been increased to achieve a shorter observation distance, while maintaining the display depth range. Because higher resolutions are required for a close observation distance and increased display viewpoints, we aimed to improve the effective resolution using almost the same pixel pitch display panel. Therefore, we built a prototype that combines a vertical red, green, blue stripe display panel and a parallax barrier with subpixel width slits. We confirmed effective resolution improvement by tiny subjective tests. This method also helps increase the depth range of the display when it is observed from a normal distance.

Industrial Scene Text Detection With Refined Feature-Attentive Network

Detecting the marking characters of industrial metal parts remains challenging due to low visual contrast, uneven illumination, corroded surfaces, and cluttered background of metal part images. Affected by these factors, bounding boxes generated by most existing methods could not locate low-contrast text areas very well. In this paper, we propose a refined feature-attentive network (RFN) to solve the inaccurate localization problem. Specifically, we first design a parallel feature integration mechanism to construct an adaptive feature representation from multi-resolution features, which enhances the perception of multi-scale texts at each scale-specific level to generate a high-quality attention map. Then, an attentive proposal refinement module is developed by the attention map to rectify the location deviation of candidate boxes. Besides, a re-scoring mechanism is designed to select text boxes with the best rectified location. To promote the research towards industrial scene text detection, we contribute two industrial scene text datasets, including a total of 102156 images and 1948809 text instances with various character structures and metal parts. Extensive experiments on our dataset and four public datasets demonstrate that our proposed method achieves the state-of-the-art performance. Both code and dataset are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/TongkunGuan/RFN</uri> .

Sequence feature extraction-based APT attack detection method with provenance graphs

Advanced persistent threat (APT) in real scenes, especially in industrial scenes, is complex and long term, but the current methods cannot effectively extract the long term relationship in the attack. An attack detection method with provenance graphs, called SeqNet, is proposed. SeqNet uses sequence feature extraction to detect APT attacks. In SeqNet, the provenance graph sequence describing the running state of the system is transformed into the feature sequence first, and then the gate recurrent unit (GRU) model is used to extract the feature of the system. The encoder-decoder model with the local attention mechanism is used to train the GRU model. Finally, the K-means clustering method is used to model the normal behavior of the system. In this study, experiments are carried out on five public datasets, such as StreamSpot, wget, shellshock, ClearScope, and CADETS, compared with the state-of-the-art methods. The method used here achieves similar or improved results on all five datasets. Experimental results show that the proposed method can detect real-life APT scenarios.

Research on Rolling Bearing Fault Diagnosis Method Based on Generative Adversarial and Transfer Learning

The diagnosis of rolling bearing faults has become an increasingly popular research topic in recent years. However, many studies have been conducted based on sufficient training data. In the real industrial scene, there are some problems in bearing fault diagnosis, including the imbalanced ratio of normal and failure data and the amount of unlabeled data being far more than the amount of marked data. This paper presents a rolling bearing fault diagnosis method suitable for different working conditions based on simulating the real industrial scene. Firstly, the dataset is divided into the source and target domains, and the signals are transformed into pictures by continuous wavelet transform. Secondly, Wasserstein Generative Adversarial Nets-Gradient Penalty (WGAN-GP) is used to generate false sample images; then, the source domain and target domain data are input into the migration learning network with Resnet50 as the backbone for processing to extract similar features. Multi-Kernel Maximum mean discrepancies (MK-MMD) are used to reduce the edge distribution difference between the data of the source domain and the target domain. Based on Case Western Reserve University′s dataset, the feasibility of the proposed method is verified by experiments. The experimental results show that the average fault diagnosis accuracy can reach 96.58%.

Attention-based multiscale denoising residual convolutional neural networks for fault diagnosis of rotating machinery

CNN-based fault diagnosis approaches have achieved promising results in improving the safety and reliability of rotating machinery. Most of the existing CNN models are developed on the assumption that the collected data is high-quality. However, since rotating machinery usually operates under fluctuating conditions, the critical pulse information of the measured vibration signals is easily submerged in noise. To promote the adaptability of CNN in noisy industrial scenes, an attention-based multiscale denoising residual convolutional neural network (AM-DRCN) is put forward in this study. First of all, a multiscale denoising module (MDM) is introduced as the basic building unit to help the network explore multiscale features and filter out irrelevant information. Then, a feature enhancement module (FEM) is leveraged to expand the receptive field and make full use of the side-out features. Further, a joint attention module (JAM) is explored to integrate the extracted features effectively. Finally, a lightweight CNN model named AM-DRCN is developed based on the above improvements. The practicality and effectiveness of AM-DRCN for monitoring machine health and stability states are verified through three case studies.

Research and Practice on Testing Methods for Mobile Robot Positioning Accuracy

To solve the problem of low positioning precision of mobile robot in conventional manual test with low test efficiency,and pose measurement based on high-precision pose measurement tools show deficiency in a low degree of automation, which fails to be directly used to evaluate test results. This paper proposes a position precision based on high precision position measuring tool automation testing methods, which can not only solve the low conventional manual test precision, and when the positioning accuracy is required to reach a higher standard, the error caused by manual measurement makes the measurement result fails to be used as the evaluation benchmark.In addition, it can automatically collect pose points in the process of repeated autonomous movement of mobile robots, and then automatically analyze the pose error when reaching a specific position point every time. It is suitable for repeated testing in industrial scenes for a long time, and achieves the purpose of improving test efficiency and productivity.

Image Translation Based Synthetic Data Generation for Industrial Object Detection and Pose Estimation

Deep learning-based methods have shown excellent potential on object detection and pose estimation with vast amounts of training data to achieve good performance. Obtaining enough comprehensive manual labeling training data is a time-consuming and error-prone task in industrial scenes, and most current time-saving synthetic data generation methods require detailed high-quality CAD models. Instead, we introduce an image-to-image translation based synthetic data generation approach, which requires only untextured CAD models and a small number of real images. The proposed approach starts with edge maps extracted by mesh model projection. And each edge map is provided with realistic appearance learned from real images through image-to-image translation to achieve patch-level realism, and then superimposed on randomly selected backgrounds with a simple Cut-and-Paste strategy. Experiments on our custom dataset and T-LESS dataset show that our synthetic data is competitive with uncomprehensive real data and gives a considerable improvement when compared to the same degree of model rendering images. In addition, the proposed approach allows for extensibility to accommodate new parts with similar texture and structure. And it shows the potential to train image translation model on only one single object and synthesize data for multiples in one take.

A comparative study on class-imbalanced gas turbine fault diagnosis

Gas turbines are widely used in various fields, and the failure of gas turbines can cause catastrophic consequences. Health condition monitoring and fault diagnosis of gas turbines can detect faults timely, avoid serious faults, and significantly reduce maintenance costs. Thus, fault diagnosis of gas turbines has great significance. Current researches on gas turbine fault diagnosis mainly focus on the case of abundant fault samples. However, fault data are very rare and the number of normal samples is much larger than the number of fault samples in the industrial scene. This class-imbalance problem widely exists but is hardly focused in the field of gas turbine fault diagnosis. Aiming to solve this problem, this paper introduces the concept of class-imbalanced learning from the machine learning field, summarizes three kinds of class-imbalance addressment methods including oversampling, undersampling, and sample weighting, and proposes a new combination method of focal loss and random oversampling for addressing class-imbalance in deep neural networks, and performs a systematic comparative study on class-imbalanced gas turbine fault diagnosis. Experimental results show that class-imbalance can seriously reduce the fault diagnosis accuracy. Through these class-imbalance addressment methods, diagnosis accuracy is greatly improved. Comparative experiments also show that the proposed combination method can obtain the best diagnosis accuracy among all the compared methods in class-imbalanced situation. Through this comparative study, a detailed guideline for improving diagnosis accuracy under class-imbalanced circumstance is provided.

Markov Transition Field Enhanced Deep Domain Adaptation Network for Milling Tool Condition Monitoring

Tool condition monitoring (TCM) is of great importance for improving the manufacturing efficiency and surface quality of workpieces. Data-driven machine learning methods are widely used in TCM and have achieved many good results. However, in actual industrial scenes, labeled data are not available in time in the target domain that significantly affect the performance of data-driven methods. To overcome this problem, a new TCM method combining the Markov transition field (MTF) and the deep domain adaptation network (DDAN) is proposed. A few vibration signals collected in the TCM experiments were represented in 2D images through MTF to enrich the features of the raw signals. The transferred ResNet50 was used to extract deep features of these 2D images. DDAN was employed to extract deep domain-invariant features between the source and target domains, in which the maximum mean discrepancy (MMD) is applied to measure the distance between two different distributions. TCM experiments show that the proposed method significantly outperforms the other three benchmark methods and is more robust under varying working conditions.

A Homogeneous Stacking Ensemble Learning Model for Fault Diagnosis of Rotating Machinery With Small Samples

As important equipment, rotating machinery has been widely used in many industrial fields. Because rotating machinery is prone to failure, its fault diagnosis will be of great significance. In the industrial scene, rotating machinery is usually in normal operation, so it is difficult to accumulate fault samples. Therefore, it will face the problem of fault diagnosis with small samples, which seriously affects the accuracy and stability of fault diagnosis. To solve the above problems, the author proposes a fault diagnosis method based on ACWGAN-GP and homogeneous stacking ensemble learning. Firstly, the method utilizes the argmax multi-class classification idea to construct multiple different training subsets. Secondly, these constructed training subsets are used to train multiple base learners based on ACWGAN-GP, Finally, the meta learner based on Softmax Regression is used to fuse these trained basic learners. So far, the complete fault diagnosis is realized. In this paper, the proposed method is applied to the gearbox data set and the bearing data set. Through a series of experiments, it is proved that this method can not only effectively solve the problem of fault diagnosis with small samples, but also effectively improve the classification accuracy and stability.

Mechatronics Equipment Performance Degradation Assessment Using Limited and Unlabeled Data

Advanced mechatronics equipment requires reliable and effective performance degradation assessment to guarantee long-term operations. Current data-driven predictions endow the operation and maintenance of mechatronic equipment flexibly and intelligently. However, the sufficient and labeled data in real industrial scenes may not be satisfied, resulting in negative impacts of overfitting and time-consuming annotations. In this article, we propose a novel prognostic model, namely unsupervised meta gated recurrent unit (UMGRU) containing a dual-cycle learning architecture with the designed clustering assignment module to deal with few-shot prognostics under unlabeled historical data. It integrates the strength of double gradient based optimizations for abstracting general degradation knowledge and offering a sensitive model status for precisely online adaptation with limited on-site data. Besides, mini-batch pseudolabels are automatically assigned within each inner cycle learning and further participate in parameter upgrades. Finally, both experimental and industrial data are used to verify the effectiveness of UMGRU.

Design of real-time target location algorithm for industrial scene

Abstract In order to meet the needs of real-time visual inspection of workpiece targets in industrial environment. In this paper an algorithm for locating workpiece based on edge feature is designed by using image information. At the same time, a model description method based on generalized Hough model and an acceleration strategy based on pyramid search are proposed to solve the problem of insufficient real-time performance of the algorithm. The algorithm is mainly divided into two stages. First, in the offline stage matching model is created. In the online stage, according to the model created in advance, the pyramid based accelerated search strategy is used to match, and the parallel computing method is adopted to calculate the similarity. Experimental results show that the algorithm has strong anti-occlusion and anti-interference ability, and can achieve real-time target positioning under any Angle and coordinates. At the same time, it is proved that the proposed acceleration strategy has better acceleration performance than the existing matching speed method.

Sound source localization of non-synchronous measurements beamforming based on the truncated nuclear norm regularization

Due to the size and aperture limitations of the microphone array, poor resolution and limited working frequency range are challenging for acoustic imaging with beamforming methods. In this work, two fast iteration algorithms, alternating direction method of multipliers (ADMM) and accelerated proximal gradient line search method (APGL) based on truncated nuclear norm regularization (TNNR), are proposed to complete the cross-spectral matrix for non-synchronous measurements beamforming. Compared with current matrix completion algorithm based on nuclear norm minimization (NNM), TNNR is regarded as a better approximation to the rank function, which means it can approach the real solution better and restore the sound sources accurately at a low signal-to-noise ratio (SNR) environment. Among the TNNR-APGL and TNNR-ADMM, the TNNR-APGL algorithm has a lower matrix completion error and a shorter calculation time. The localization results of TNNR-ADMM method are more robust to the number of measurements. The experiment results show that TNNR methods greatly improve the resolution of conventional beamforming (CBF) and suppress the sidelobes under low SNR conditions, which indicates that the non-synchronous measurements beamforming based on TNNR methods has the potential to be applied in the industrial scene and low SNR environment.

Experimental and numerical investigations on heat transfer and flow behavior of flow blockage in narrow rectangular channel with protrusions

The safety analysis of flow blockage accident is significant for plate-type fuel reactor, especially swelling of cladding, which is resulted from irradiation damage in high burnup condition and needs to be paid more attention. In this study, experimental and numerical researches have been conducted to study the convective heat transfer and flow structure within narrow rectangular channel, where exists different numbers and arrangements of spherical crown protrusions. Compared with the experimental data of single protrusion condition, the overall heat transfer capacity even deteriorates more than 20% when protrusions are aligned in streamwise direction. In contrast, the global Nu is nearly 10% higher when three protrusions are located in spanwise direction. Conclusions can be drawn that existence of protrusion can also significantly change convective flow condition in narrow channel, and the laminar flow is altered to transitional/quasi-turbulent even though Re is only nearly 1600. Besides, the local information, such as flow boundary layer separation, low-speed recirculation and vortex evolution etc., can also be acquired by the Computational Fluid Dynamics (CFD) method comprehensively. Combined with experimental data and numerical results, the distinct heat transfer behaviors and flow structures for different flow blockage conditions can be explained in detail. In summary, conclusions about the flow and thermal processes influenced by protrusions in narrow rectangular channel are fundamental to safety analysis and design criteria of plate-type fuel assembly for flow blockage condition. Besides, it also can be applied in other industry scene during convective laminar flow within narrow rectangular channel.