Complex Industrial Environments Research Articles

AUTOMATIC INTEGRATION OF LASER SCANNING AND PHOTOGRAMMETRIC POINT CLOUDS: FROM ACQUISITION TO CO-REGISTRATION

Abstract. Laser scanning systems have been developed to capture very high-resolution 3D point clouds and consequently acquire the object geometry. This object measuring technique has a high capacity for being utilized in a wide variety of applications such as indoor and outdoor modelling. The Terrestrial Laser Scanning (TLS) is used as an important data capturing measurement system to provide high quality point cloud from industrial or built-up environments. However, the static nature of the TLS and complexity of the industrial sites necessitate employing a complementary data capturing system e.g. cameras to fill the gaps in the TLS point cloud caused by occlusions which is very common in complex industrial areas. Moreover, employing images provide better radiometric and edge information. This motivated a joint project to develop a system for automatic and robust co-registration of TLS data and images directly, especially for complex objects. In this paper, the proposed methods for various components of this project including gap detection from point cloud, calculation of initial image capturing configuration, user interface and support system for the image capturing procedures, and co-registration between TLS point cloud and photogrammetric point cloud are presented. The primarily results on a complex industrial environment are promising.

An IoT‐based human detection system for complex industrial environment with deep learning architectures and transfer learning

Artificial intelligence (AI), combined with the Internet of Things (IoT), plays a beneficial role in various fields, including intelligent surveillance applications. With IoT and 5G advancement, intelligent sensors, and devices in the surveillance environment collect large amounts of data in the form of videos and images. These collected data require intelligent information processing solutions, help analyze the recorded videos and images to detect and identify various objects in the scene, particularly humans. In this study, an automated human detection system is presented for a complex industrial environment, in which people are monitored/detected from a top view perspective. A top view is usually preferred because it can provide sufficient coverage and enough visibility of a scene. This study demonstrates the applications, efficiency, and effectiveness of deep learning architectures, that is, Faster Region Convolutional Neural Network (Faster R-CNN), Single Shot MultiBox Detector (SSD), and You Only Look Once (YOLOv3), with transfer learning. Experimental results reveal that with additional training and transfer learning, the performance of all detection architectures is significantly improved. The detection results are also compared using the same data set. The deep learning architectures achieve promising results with maximum true-positive rate of 93%, 94%, and 94% for Faster-RCNN, SSD, and YOLOv3, respectively. Furthermore, a detailed study is performed on output results that highlight challenges and probable future trends.

Robotic Waste Sorting Technology: Toward a Vision-Based Categorization System for the Industrial Robotic Separation of Recyclable Waste

The use of robots in waste processing plants can significantly improve the processing of recyclables. Such robots need sophisticated visual and manipulation skills to be able to work in the extremely heterogeneous, complex, and unpredictable waste sorting industrial environment. This article considers the implementation of an autonomous robotic system for the categorization and physical sorting of recyclables according to material types. In particular, it focuses on the development of a low-cost computer vision module based on deep learning technologies to identify and sort items. To facilitate further research endeavors, the data set of recyclable images and a group of image processing scripts for object identification, masking, and synthetic placement against multiple backgrounds are available in an open source GitHub repository (https://github.com/kskmar/ReSort-IT.git). The deep-trained computer vision module is integrated with a robotic system that undertakes the physical separation of recyclables. The composite system is deployed in a waste processing plant, where it is successfully assessed in recyclable sorting under difficult and demanding industrial conditions.

Reliable Personnel Positioning in Industrial Environments Based on Improved Adaptive EKF With Random Packet Loss

In the complex industrial environment, random packet loss may occur in the process of sensor data transmission. The traditional Extended Kalman Filter (EKF) algorithm will reduce the estimation accuracy, even lead to the divergence of the estimator. To solve these problems, an improved adaptive extended Kalman filter (IAEKF) is proposed to estimate the covariance matrix of process noise adaptively. At the same time, the forgetting factor of strong tracking filter (STF) is introduced to improve the robustness of the algorithm in the case of random packet loss. Simulation results show that IAEKF algorithm can effectively reduce the personnel positioning error in the case of random packet loss, and has better localization accuracy, meeting the requirements of industrial environment.

Anomaly Detection Based on Multidimensional Data Processing for Protecting Vital Devices in 6G-Enabled Massive IIoT

As a result of the increasing deployment of Industrial-Internet-of-Things (IIoT) architectures, large volumes of multidimensional data are continuously generated. An important issue with these data is that higher dimensionality increases the degree of fragmentation. Furthermore, data sets collected by IIoT nodes often display outliers, which are usually caused by anomalous events or errors. These outliers contain considerable valuable information, which prevent the normal operation of the system. Thus, methodologies are able to quantify the obtained information to protect the high priority IIoT nodes, are crucial. This study aims at developing such a method driven by sixth-generation (6G) networks. The proposed algorithm uses a multidimensional data relationship diagram to characterize the spatiotemporal correlations among heterogeneous data. Then, an autoregressive exogenous model is used to eliminate the effects of noise on sensor data, and to help in detecting anomalies. Finally, the algorithm produces a Cumulative Coefficient of Value (CCoV), to identify high-value sensing devices and enable massive Internet of Things (IoT) with 6G-using the characteristic patterns hidden within the data. The experimental results demonstrate that the proposed method can effectively handle the effects of the ubiquitous interference noise in complex industrial environments. Moreover, the method yields effective anomaly detection and compensates for some of the shortcomings in traditional methods.

A novel topology optimization of coverage-oriented strategy for wireless sensor networks

Aiming at the key optimization problems of wireless sensor networks in complex industrial application environments, such as the optimum coverage and the reliability of the network, a novel topology optimization of coverage-oriented strategy for wireless sensor networks based on the wolf pack algorithm is proposed. Combining the characteristics of topology structure of wireless sensor networks and the optimization idea of the wolf pack algorithm redefines the group’s wandering and surprise behavior. A novel head wolf mutation strategy is proposed, which increases the neighborhood search range of the optimal solution, enhances the uniformity of wolf pack distribution and the ergodicity ability of the wolf pack search, and greatly improves the calculation speed and the accuracy of the wolf pack algorithm. With the same probability, the cluster heads are randomly selected periodically, and the overall energy consumption of wireless sensor networks is evenly distributed to the sensor node to realize the balanced distribution of the data of the member nodes in the cluster and complete the design of the topology optimization of wireless sensor networks. Through algorithm simulation and result analysis, compared with the particle swarm optimization algorithm and artificial fish swarm algorithm, the wolf swarm algorithm shows its advantages in terms of the residual energy of the sensor node, the average transmission delay, the average packet delivery rate, and the coverage of the network. Among them, compared with the particle swarm optimization algorithm and artificial fish swarm algorithm, the remaining energy of nodes has increased by 9.5% and 15.5% and the average coverage of the network has increased by 10.5% and 5.6%, respectively.

Attention-based encoder-decoder networks for workflow recognition

Behavior recognition is a fundamental yet challenging task in intelligent surveillance system, which plays an increasingly important role in the process of “Industry 4.0”. However, monitoring the workflow of both workers and machines in production procedure is quite difficult in complex industrial environments. In this paper, we propose a novel workflow recognition framework to recognize the behavior of working subjects based on the well-designed encoder-decoder structure. Namely, attention-based workflow recognition framework, termed as AWR. To improve the accuracy of workflow recognition, a temporal attention cell (AttCell) is introduced to draw dynamic attention distribution in the last stage of the framework. In addition, a Rough-to-Refine phase localization model is exploited to improve localization accuracy, which can effectively identify the boundaries of a specific phase instance in long untrimmed videos. Comprehensive experiments indicate a 1.4% mAP@IoU= 0.4 boost on THUMOS’14 dataset and a 3.4% mAP@IoU= 0.4 boost on hand-crafted workflow dataset detection challenge compared to the advanced GTAN pipeline respectively. More remarkably, the effectiveness of the workflow recognition system is validated in a real-world production scenario.

A robust identification method for nonferrous metal scraps based on deep learning and superpixel optimization

End-of-life vehicles (ELVs) provide a particularly potent source of supply for metals. Hence, the recycling and sorting techniques for ferrous and nonferrous metal scraps from ELVs significantly increase metal resource utilization. However, different kinds of nonferrous metal scraps, such as aluminium (Al) and copper (Cu), are not further automatically classified due to the lack of proper techniques. The purpose of this study is to propose an identification method for different nonferrous metal scraps, facilitate the further separation of nonferrous metal scraps, achieve better management of recycled metal resources and increase sustainability. A convolutional neural network (CNN) and SEEDS (superpixels extracted via energy-driven sampling) were adopted in this study. To build the classifier, 80 training images of randomly chosen Al and Cu scraps were taken, and some practical methods were proposed, including training patch generation with SEEDS, image data augmentation and automatic labelling methods for enormous training data. To obtain more accurate results, SEEDS was also used to optimize the coarse results obtained from the pretrained CNN model. Five indicators were adopted to evaluate the final identification results. Furthermore, 15 test samples concerning different classification environments were tested through the proposed model, and it performed well under all of the employed evaluation indexes, with an average precision of 0.98. The results demonstrate that the proposed model is robust for metal scrap identification, which can be expanded to a complex industrial environment, and it presents new possibilities for highly accurate automatic nonferrous metal scrap classification.

Lytic polysaccharide monooxygenases and other histidine-brace copper proteins: structure, oxygen activation and biotechnological applications.

Lytic polysaccharide monooxygenases (LPMOs) are mononuclear copper enzymes that catalyse the oxidative cleavage of glycosidic bonds. They are characterised by two histidine residues that coordinate copper in a configuration termed the Cu-histidine brace. Although first identified in bacteria and fungi, LPMOs have since been found in all biological kingdoms. LPMOs are now included in commercial enzyme cocktails used in industrial biorefineries. This has led to increased process yield due to the synergistic action of LPMOs with glycoside hydrolases. However, the introduction of LPMOs makes control of the enzymatic step in industrial stirred-tank reactors more challenging, and the operational stability of the enzymes is reduced. It is clear that much is still to be learned about the interaction between LPMOs and their complex natural and industrial environments, and fundamental scientific studies are required towards this end. Several atomic-resolution structures have been solved providing detailed information on the Cu-coordination sphere and the interaction with the polysaccharide substrate. However, the molecular mechanisms of LPMOs are still the subject of intense investigation; the key question being how the proteinaceous environment controls the copper cofactor towards the activation of the O-O bond in O2 and cleavage of the glycosidic bonds in polysaccharides. The need for biochemical characterisation of each putative LPMO is discussed based on recent reports showing that not all proteins with a Cu-histidine brace are enzymes.

A Low Discrepancy Heuristic Evolution ELM and Ground Effect Theory-Based Serial Hybrid Soft Sensor Model of Floating Height in Air Cushion Furnace

In air cushion furnace, the floating height is a key process parameter which greatly affects the quality and production efficiency of high quality metal strips. However, the floating height is hard to be collected in the complex and abominable industry environment. Furthermore, due to the flow field characteristics, some important process variables are difficult to accurately calculate by traditional mechanism modeling methods. In order to accurately predict the floating height, firstly, a low discrepancy heuristic evolution ELM and ground effect theory based serial hybrid soft sensor model is proposed, which constituted by a mechanism model and two data driven models. Secondly, based on the force equilibrium equation and ground effect theory, the mechanism model is constructed, which describes the relationship between the floating height and the process variables including the jet impact angle. Thirdly, a low discrepancy heuristic evolution ELM is proposed as the data driven model to predict the jet impinging angle. In the data driven model, the novel dual mutation strategies collaboration differential evolution is proposed to guarantee the low discrepancy and physical applicability of data driven model. The effectiveness of the proposed method was validated on the self-developed air cushion experiment platform and got desirable experimental results. The research lays an important foundation for the successful implementation of monitoring and control of the strip floating process.

Taguchi based Case study in the automotive industry: nonconformity decreasing with use of Six Sigma methodology

In this study, we applied a conceptual Six Sigma/design of experiment hybrid framework that aims to integrate the Taguchi method and Six Sigma for process improvement in a complex industrial environment. In this context, the Six Sigma methodology was employed on a company operating within the automotive industry to improve a manufacturing process which caused a customer complaint within the company. Studies employing the Taguchi experiment design usually focus on a single variable and neglect the effects of adjustments on remaining quality characteristics. In this study, a multi-response Taguchi design of experiment was preferred, and all of the quality characteristics were taken into account. In our study, define, measure, analysis, improve and control phases were used to reduce the nonconformity rate from 23.940 percent (baseline) to 0.049 percent. As a result of implementing Six Sigma, the sigma level increased from 2.21 (baseline) to 4.80.

Automatic Detection and Location of Weld Beads With Deep Convolutional Neural Networks

Welding quality detection is a critical link in modern manufacturing, and the weld bead location is a prerequisite for the high-precision assessment of welding quality. It is generally necessary for weld bead detection to be accomplished in the context of complex industrial environments. However, conventional detection and location methods based on specific detection conditions or prior knowledge lack accuracy and adaptability. To precisely detect and locate the weld beads in real industrial environments, a novel weld bead detection and location algorithm is proposed based on deep convolutional neural networks. Because there is no open data set of weld beads and the samples in real industrial applications are insufficient for effective model training of the deep convolutional neural network, a novel data augmentation method based on a deep semantic segmentation network is proposed to increase the sample diversity and enlarge the data set. Then, a dynamic sample updating strategy is put forward to cover more welding situations. Finally, faced with the weak-feature and weak-texture characteristics of weld beads, a simplified YOLOV3 model is proposed to realize end-to-end weld bead location. Experiments demonstrate that the proposed method could effectively satisfy the robustness and precision requirements for weld bead detection and location combined with a deep semantic segmentation network and simplified YOLOV3 model.

Integrating the HACCP and SPC for hazard control and process improvement: a case of pharmaceutical industry

Purpose The purpose of this study is to integrate two well-established frameworks with an aim to reduce the unwanted rejection rate recurring in the drug production process in pharmaceutical industries. The effectiveness of an integrated framework has been demonstrated by a real-time case study in a complex industrial environment, providing a platform for quality tools application in the pharmaceutical industry. Design/methodology/approach The hazard analysis and critical control points (HACCP) provided a basic framework for hazard analysis and its blending with statistical process control (SPC) aided in data-driven decision-making. The extensive brainstorming and Pareto analysis helped to identify potential critical-to-quality characteristics followed by SPC, x¯ and R charts, histograms and Cp and Cpk analysis to spot the critical control point. The fishbone diagram led to the extraction of the leading cause behind the identified problem. Then, based on recommended corrective actions, control limits were adjusted and the process was brought into control. Finally, a product-based cost analysis is also performed to illustrate the financial impact resulting from the proposed method’s successful implementation. Findings The integrated framework is applied to a drug production process which has a higher rejection rate (3%) because of the non-conformities. Based on Pareto analysis, potential failure causes were classified and prioritised as inappropriate composition (2.54%), packaging (0.35%), out-of-specification (0.069%), equipment failure (0.022%), input materials (0.018%) and miscellaneous (0.002%). It is found that 84% of the total rejection rate is contributed by inappropriate composition. After taking corrective actions, it is observed that the proposed method has helped to reduce the process rejection rate significantly (2.54-0.82%). In the monetary terms, 2.50% decline per unit costs is noted in this case study work. The proposed integrated framework’s success is further motivating other pharmaceutical industries to implement and expand it to other processes. Originality/value The case study is an attempt to contribute to the existing literature of quality management in pharmaceutical industries. In particular, it is a novel example to introduce the simple and user-friendly SPC tool into well-established HACCP framework to enhance its effectiveness in hazard identification. The case study results motivate managers to adopt quality techniques for achieving a higher quality standard and operational excellence.

Edge Computing-Enabled Deep Learning for Real-time Video Optimization in IIoT

Real-time multimedia applications have gained immense popularity in the industrial Internet of Things (IIoT) paradigm. Due to the impact of the complex industrial environment, the transmission of video streaming is usually unstable. In the duration of a low bandwidth transmission, existing optimization methods often reduce the original resolution of some frames in a random way to avoid the video interruption. If the key frames with some important content are selected to be transmitted with a low resolution, it will greatly reduce the effect of industrial supervision. In view of this challenge, a real-time video streaming optimization method by reducing the number of video frames transmitted in the IIoT environment is proposed. Concretely, a deep learning-based object detection algorithm is recruited to effectively select the key frames in our method. The key frames with the original resolution will be transmitted along with audio data. As some nonkey frames are selectively discarded, it is helpful for smooth network transmitting with fewer bandwidth requirements. Moreover, we employ edge servers to run the object detection algorithm, and adjust video transmission flexibly. Extensive experiments are conducted to validate the effectiveness, and dependability of our method.

Intelligent Signal Classification in Industrial Distributed Wireless Sensor Networks Based Industrial Internet of Things

In industrial sensor networks, complex industrial environments may be encountered leading to a mix of signals of different types. Complicated interference caused by mixed signals on industrial equipments may significantly degrade the classification rate of signals, which may result in a long training time in order to extract features. In addition, with limited channel resources, it is difficult to make the global optimal decision in industrial distributed wireless sensor networks. To address this problem, a signal classification method using feature fusion is proposed for industrial Internet of Things in this article. In the proposed method, the received signals of nodes are processed by frequency reduction and sampling pretreatment, based on which intelligent representations of signals are obtained. Using federated learning, the data samples are trained with the feature fusion network. Moreover, the trained deep learning network is used on each sensor node to classify signals, the results of which will be transmitted to aggregation center. In the aggregation center, the improved evidence theory method is used to aggregate the recognition results of each sensor node to achieve the final classification. Simulation shows that the proposed method has excellent classification performances. Notably, it is not required for the proposed method to transmit signals from nodes to the aggregation center, which could effectively protect the privacy of industrial information.

Double-level adversarial domain adaptation network for intelligent fault diagnosis

Deep neural networks have been widely studied in the field of mechanical fault diagnosis with the rapidity of intelligent manufacturing and industrial big data, however, attractive performance gains usually come from a premise that source training data and target test data have the same distribution. Unfortunately, this assumption is generally untenable in practice due to changeable working conditions and complex industrial environment. To address this issue, a double-level adversarial domain adaptation network (DL-ADAN) is presented for cross-domain fault diagnosis, which is able to bridge the divergences between the source and target domains. Specifically, the proposed diagnostic framework is composed of a feature extractor based on deep convolutional network, a domain discriminator and two label classifiers, which conducts two minimax adversarial games. In the first adversarial stream, the feature extractor and domain discriminator game with each other to achieve domain-level alignment from a global perspective. On the other line, the extractor and two classifiers are against each other to conduct class-level alignment, in which Wasserstein discrepancy is used to detect outlier target samples. As a result, the extractor can learn transferable discriminative features for accurate fault diagnosis. Extensive diagnostic experiments are constructed for performance analysis and several state of the art diagnostic methods are selected for comparative study. The comprehensive results demonstrate the effectiveness and superiority of the proposed method.

Visual Saliency Detection for Over-Temperature Regions in 3D Space via Dual-Source Images.

To allow mobile robots to visually observe the temperature of equipment in complex industrial environments and work on temperature anomalies in time, it is necessary to accurately find the coordinates of temperature anomalies and obtain information on the surrounding obstacles. This paper proposes a visual saliency detection method for hypertemperature in three-dimensional space through dual-source images. The key novelty of this method is that it can achieve accurate salient object detection without relying on high-performance hardware equipment. First, the redundant point clouds are removed through adaptive sampling to reduce the computational memory. Second, the original images are merged with infrared images and the dense point clouds are surface-mapped to visually display the temperature of the reconstructed surface and use infrared imaging characteristics to detect the plane coordinates of temperature anomalies. Finally, transformation mapping is coordinated according to the pose relationship to obtain the spatial position. Experimental results show that this method not only displays the temperature of the device directly but also accurately obtains the spatial coordinates of the heat source without relying on a high-performance computing platform.

Map Recovery and Fusion for Collaborative Augment Reality of Multiple Mobile Devices

The map recovery and fusion is a key issue in the application of large scale and long-term augmented reality (AR) scenarios. However, they are still not addressed well in an efficient and precise way, especially for complex industrial environments. In this article, we propose a map recovery and fusion strategy based on vision-inertial simultaneous localization and mapping. We first develop a heuristic strategy that can fast search and match map points among multiple maps, and can be used for efficient map fusion. For map recovery, we leverage the inertial sensors for short time motion estimation, and transform the previous lost map to the current map. Based on this strategy, a novel framework for collaborative AR is implemented and can parallelly run in multiple mobile devices in real time. Extensive experiments have been carried out on a public data set, and the results show that the proposed method can recovery and fuse multiple maps with high completeness and precision.

Recrystallization and coalescence kinetics of TiO2 and ZnO nano-catalysts towards enhanced photocatalytic activity and colloidal stability within slurry reactors

Advanced oxidation processes rely on the development of stable photocatalytic materials, offering specific band-gap attained upon reaching appropriate crystalline phases. A key research gap is related to the recrystallization of nano-catalysts and their impact on performance. This paper describes the systematic preparation of TiO2 and ZnO nanoparticles by hydrothermal-assisted sol-gel method followed by systematic calcination steps with the aim to shed light on the recrystallization process to engineer higher photocatalytic activity. Spherical anatase-TiO2 nanoparticles with a 20 nm diameter size present higher photocatalytic activity than 70 nm of ZnO NPs calcined at 500 °C. A photocatalytic yield of 84% within 30 min of irradiation for the degradation of model dyes was observed for the titania particles, which were also less sensitive to a gglomeration, a key challenge when designing slurry reactors. The variation in zeta potential of TiO2 and ZnO with pH exhibited isoelectric points (IEP) in aqueous media at 5.1 and 6.5, respectively, suggesting amphoteric behaviors while X ray photo-electron spectroscopy and diffusive reflectance spectroscopy data were used to characterize the changes in surface vacancies and the band gap of the materials. These data open the door to the development of advanced oxidation processes in complex industrial environments.

Complete calibration of a structured light stripe vision sensor through a single cylindrical target

In a complex industrial environment, it is not appropriate to use the traditional Zhang's calibration method to shoot multiple checkerboard images in different positions, and due to the contour characteristics of the cylindrical target, camera calibration and line-structured calibration can be simultaneously performed. Therefore, a complete calibration method for a line-structured light system based on a single cylindrical target is proposed. The calibration process is as follows: Placing the cylindrical target at least once and illuminate it with the light strip of the laser projector. The vision sensor captures the target image and then extracts the laser stripes and contour images of the cylindrical target. According to the contour characteristics of the cylindrical target and the definition of the collinear vector, the extracted contour image is used to solve the internal parameters of the camera. Because the line-structured light plane and cylindrical target intersection line in the axial of cylindrical is a standard circle, the equation of the light plane is established on the camera coordinate system. The effectiveness of the method is verified by simulation and physical experiments. The experimental results show that the RMS errors of the 30 distance is 0.07 mm. The proposed method does not need to rely on the constraints of multiple images and is simple to operate, meeting the requirements of industrial 3D measurement technology.