Complex Pattern Recognition Research Articles

Extending the Tsetlin Machine With Integer-Weighted Clauses for Increased Interpretability

Building models that are both interpretable and accurate is an unresolved challenge for many pattern recognition problems. In general, rule-based and linear models lack accuracy, while deep learning interpretability is based on rough approximations of the underlying inference. However, recently, the rule-based Tsetlin Machines (TMs) have obtained competitive performance in terms of accuracy, memory footprint, and inference speed on diverse benchmarks (image classification, regression, natural language understanding, and game-playing). TMs construct rules using human-interpretable conjunctive clauses in propositional logic. These, in turn, are combined linearly to solve complex pattern recognition tasks. This paper addresses the accuracy-interpretability challenge in machine learning by introducing a TM with integer weighted clauses - the Integer Weighted TM (IWTM). The intent is to increase TM interpretability by reducing the number of clauses required for competitive performance. The IWTM achieves this by weighting the clauses so that a single clause can replace multiple duplicates. Since each TM clause is formed adaptively by a Tsetlin Automata (TA) team, identifying effective weights becomes a challenging online learning problem. We solve this problem by extending each team of TA with another kind of automaton: the stochastic searching on the line (SSL) automaton. We evaluate the performance of the new scheme empirically using five datasets, along with a study of interpretability. On average, IWTM uses 6.5 times fewer literals than the vanilla TM and 120 times fewer literals than a TM with real-valued weights. Furthermore, in terms of average memory usage and F1-Score, IWTM outperforms simple Multi-Layered Artificial Neural Networks, Decision Trees, Support Vector Machines, K-Nearest Neighbor, Random Forest, Gradient Boosted Trees (XGBoost), Explainable Boosting Machines (EBMs), as well as the standard and real-value weighted TMs. IWTM finally outperforms Neural Additive Models on Fraud Detection and StructureBoost on CA-58 in terms of Area Under Curve, while performing competitively on COMPAS.

Automatic system design for flange surface quality inspection with a bionic motion-vision paradigm

ABSTRACT Flanges are used to make connections. Their surface quality determines the joint mechanical strength and characteristics, while their defects lead to device failure. Complex surfaces and surface flaws pose challenges to automatic optical inspection (AOI) and conventionally, visual and manual inspection is employed. The aim of this study is to develop an automatic defect detection system for the flange surface, an operational pipeline with a bionic motion-vision mechatronic system, which mimics manual detection. Three methods are proposed as part of the system. The visual-detection-simulating sensing method, mimicking humans defect detection under horizontal light, is proposed to produce images of the defects on multifaceted reflective-metal surfaces. The human-operation simulation method, mimicking human defects detection under different light angles, is designed to optimize the visual perspective angles. The intelligent decision mechanism, mimicking the way human attention is focused, is proposed to solve complex pattern recognition problems. Based on this approach, an automated inspection machine, including posture adjustment, lighting, and imaging, for flange surface quality is developed. An intelligent inspection system with defect recognition algorithms is deployed. The smallest width of surface defects detected is 0.1 mm. The proposed apparatus design paradigm provides a general solution for flaw-detection on non-planar, complex surface workpieces.

Deformable Convolutional Networks for Efficient Mixed-Type Wafer Defect Pattern Recognition

Defect pattern recognition (DPR) of wafer maps is critical for determining the root cause of production defects, which can provide insights for the yield improvement in wafer foundries. During wafer fabrication, several types of defects can be coupled together in a piece of wafer, it is called mixed-type defects DPR. To detect mixed-type defects is much more complicated because the combination of defects may vary a lot, from the type of defects, position, angle, number of defects, etc. Deep learning methods have been a good choice for complex pattern recognition problems. In this article, we propose a deformable convolutional network (DC-Net) for mixed-type DPR (MDPR) in which several types of defects are coupled together in a piece of wafer. A deformable convolutional unit is designed to selectively sample from mixed defects, then extract high-quality features from wafer maps. A multi-label output layer is improved with a one-hot encoding mechanism, which decomposes extract mixed features into each basic single defect. The experiment results indicate that the proposed DC-Net model outperforms conventional models and other deep learning models. Further results of the interpretable analysis reveal that the proposed DC-Net can accurately pinpoint the defects areas of wafer maps with noise points, which is beneficial for mixed-type DPR problems.

P975Composite electroanatomical maps locate rapid activity within low voltage zones in persistent AF

Abstract Funding Acknowledgements Our research group receives an educational grant from Abbott Inc. Introduction. Outcomes from catheter ablation of persistent AF (psAF) are not favourable. The two prevailing major directions to improve success are left atrial (LA) substrate ablation, and non pulmonary vein driver ablation. In LA substrate ablation guided by intracardiac voltage, there is debate on the most fitting mapping rhythm and the appropriate cut offs for low voltage zones (LVZ). Non pulmonary vein driver ablation requires extensive experience and relies on complex pattern recognition by the operator, introducing subjectivity, that may lead to reduced reproducibility. AF drivers have been shown to localise to LVZs. We propose an objective, patient-tailored method of identifying rapid activity within LVZs to locate drivers of psAF. Methods. Eleven patients (61 ± 10.8 years of age, 9 male) undergoing first time catheter ablation for psAF were included. 3D maps were collected with a double spiral 20 pole catheter, in non-cardiac triggered mode, recording 8s segments at each bipole. Mean AF voltage (AFV) a AF cycle length (AFCL) was calculated for each 8s segment using automated algorithms. Grades of rapid activity and low voltage were defined as the 10th 20th and 30th percentile of all collected points within a patient. Percentile-matched composite LVZ-ARA maps were created on a research platform. Results. Mean LVZ percentage of the total mapped area was 4.67 ± 2.4%, 13.95 ± 3.8%, 23.81 ± 5.7% for the 10th, 20th and 30th percentiles respectively (Table 1). Mean, percentile matched LVZ-ARA overlap area percentage of the total mapped area was 0.3 ± 0.25% (10th-10th), 0.86 ± 0.58 (20th-20th), 3.1 ± 1.9% (30th-30th). ARAs represented a small proportion of all LVZs. Location of overlap areas differed significantly between patients and were marked with colours. Multi-colour areas including purple represent LVZ, multi-colour areas excluding purple, show LVZ-ARA overlap (examples in Fig 1). Conclusion. Analysis of LVZ-ARA overlap by mean AFV and AFCL provides an objective method of identifying potential drivers that localise to LVZs. The identified overlap areas constituted small, occasionally disparate areas within the LVZ of the LA. By adjusting the AFCL and AFV percentiles, the overlap areas can be tailored at the operator’s discretion, maintaining reproducible, objective decision making, without the need for complex pattern recognition. If ablation is planned, established techniques can be used to target the overlap areas, such as homogenisation or transection and connection to anatomical or ablative non-conductive tissues. AFCL 10th AFCL 20th AFCL 30th AFV 10th AFV 20th AFV 30th All patients 128 ± 13 ms 144 ± 10 ms 150 ± 9 ms 0.15 ± 0.02 mV 0.19 ± 0.03 mV 0.24 ± 0.04 mV Mean values of percentile cut offs. AFCL: AF cycle length; AFV: AF voltage Abstract Figure. Fig 1

Sparse Deep Neural Network Optimization for Embedded Intelligence

Deep neural networks become more popular as its ability to solve very complex pattern recognition problems. However, deep neural networks often need massive computational and memory resources, which is main reason resulting them to be difficult efficiently and entirely running on embedded platforms. This work addresses this problem by saving the computational and memory requirements of deep neural networks by proposing a variance reduced (VR)-based optimization with regularization techniques to compress the requirements of memory of models within fast training process. It is shown theoretically and experimentally that sparsity-inducing regularization can be effectively worked with the VR-based optimization whereby in the optimizer the behaviors of the stochastic element is controlled by a hyper-parameter to solve non-convex problems.

An Overview on Horizon Detection Methods in Maritime Video Surveillance

The interest in video surveillance has been increasing in the fields of maritime industry in the past decade. Maritime transportation system is a vital part of the world’s economy and the extent of global ship traffic is increasing. This trend encourages the development of intelligent surveillance systems in the maritime zone. The development of intelligent surveillance systems includes sensor and data fusion, which incorporates multispectral and multisensory data to replace the traditional approach with radars only. Video cameras are widely used since they capture images of greater resolution than most sensor systems. Also, combined with video analytics they provide sensors with high capability, complex pattern recognition analytics, and multiple variables for the decision making process. In this paper, an overview of a small part of the system is presented – horizon detection.

WHY DEEP LEARNING PERFORMS BETTER THAN CLASSICAL MACHINE LEARNING?

During the last years, deep learning techniques have demonstrated their capability to outperform traditional machine learning methods in completing complex pattern recognition tasks. In this article we will try to explain the reasons behind this.

Digital elevation model-based convolutional neural network modeling for searching of high solar energy regions

The prediction of potential solar energy at candidate sites plays a key role in the search for high solar energy regions to accommodate solar photovoltaic facilities. However, the estimation of solar irradiation can be inaccurate when site conditions are not similar to the ones at observation stations. In particular, regional effects caused by adjacent terrains have rarely been modeled in the estimation process. This study thus presents a digital elevation map-based convolutional neural network modeling method for the prediction of annual solar irradiation under clear-sky conditions. Using map data as an input, the sole impact of neighboring topography on available solar energy can be understood at a large scale. Specifically, with elevation maps and corresponding solar irradiation maps, the mean of solar irradiation values for each map is calculated. This mean value is then used as an output to identify the non-linear and hidden relationship between such values and the input maps in training processes. The proposed network model uses terrain map datasets and hence enables the recognition and learning of complex topographic features and patterns in the datasets. As a result, the network model has a mean absolute percent error of 0.470% for testing datasets. This implies that the topographic patterns on a large scale may coincide with variations of available solar irradiation prior to the effects of weather. Thus, the integration of raster map data with solar energy prediction has emerged to systematically learn the terrain patterns in order to predict the annual amount of solar irradiation in a region of interest. This approach can aid the determination of suitable locations for the installation of solar panels.

Deep Neural Network Technique for High-Dimensional Microwave Modeling and Applications to Parameter Extraction of Microwave Filters

This article introduces the deep neural network method into the field of high-dimensional microwave modeling. Deep learning is nowadays highly successful in solving complex and challenging pattern recognition and classification problems. This article investigates the use of deep neural networks to solve microwave modeling problems that are much more challenging than that solved by the previous shallow neural networks. The most commonly used activation function in the existing deep neural network is the rectified linear unit (ReLU), which is a piecewise hard switch function. However, such a ReLU is not suitable for microwave modeling where the input–output relationships are smooth and continuous. In this article, we propose a new deep neural network to perform high-dimensional microwave modeling. A smooth ReLU is proposed for the new deep neural network. The proposed deep neural network employs both the sigmoid function and the smooth ReLU as activation functions. The new deep neural network can represent the smooth input–output relationship that is required for microwave modeling. An advanced three-stage deep learning algorithm is proposed to train the new deep neural network model. This algorithm can determine the number of hidden layers with sigmoid functions and those with smooth ReLUs in the training process. It can also overcome the vanishing gradient problem for training the deep neural network. The proposed deep neural network technique can solve microwave modeling problems in a higher dimension than the previous neural network method, i.e., shallow neural network method. Two high-dimensional parameter-extraction modeling examples of microwave filters are presented to demonstrate the proposed deep neural network technique.

Fully invariant generalized Hough transform by out-of-focus multiview sensing with pupil array

Recognition of geometrical shapes in real-time and fully invariant (i.e., invariant under changes in position, scale, and orientation) is a demanding task in automated image analysis. In particular, the generalized Hough transform (GHT) is a well-known algorithm for the recognition of complex patterns out of edge binary images even with disconnected boundaries or corrupted by noise. In this work we present a space multiplexed optical implementation of the GHT which, by exploiting the redundancy derived from multiview sensing of a two-dimensional image and its out- of-focus capture with an adequate pupil array, allows us to obtain in a single shot the GHT of this image invariant to target shift, scale, and orientation. Experimental validation of the working principle is presented, along with an assessment of the robustness of the system against noise in the input.

An Attention-Based Spiking Neural Network for Unsupervised Spike-Sorting.

Bio-inspired computing using artificial spiking neural networks promises performances outperforming currently available computational approaches. Yet, the number of applications of such networks remains limited due to the absence of generic training procedures for complex pattern recognition, which require the design of dedicated architectures for each situation. We developed a spike-timing-dependent plasticity (STDP) spiking neural network (SSN) to address spike-sorting, a central pattern recognition problem in neuroscience. This network is designed to process an extracellular neural signal in an online and unsupervised fashion. The signal stream is continuously fed to the network and processed through several layers to output spike trains matching the truth after a short learning period requiring only few data. The network features an attention mechanism to handle the scarcity of action potential occurrences in the signal, and a threshold adaptation mechanism to handle patterns with different sizes. This method outperforms two existing spike-sorting algorithms at low signal-to-noise ratio (SNR) and can be adapted to process several channels simultaneously in the case of tetrode recordings. Such attention-based STDP network applied to spike-sorting opens perspectives to embed neuromorphic processing of neural data in future brain implants.

Rank Based Pseudoinverse Computation in Extreme Learning Machine for large Datasets

Extreme Learning Machine (ELM) is an efficient and effective least-square-based learning algorithm for classification, regression problems based on single hidden layer feed-forward neural network (SLFN). It has been shown in the literature that it has faster convergence and good generalization ability for moderate datasets. But, there is great deal of challenge involved in computing the pseudoinverse when there are large numbers of hidden nodes or for large number of instances to train complex pattern recognition problems. To address this problem, a few approaches such as EM-ELM, DF-ELM have been proposed in the literature. In this paper, a new rank-based matrix decomposition of the hidden layer matrix is introduced to have the optimal training time and reduce the computational complexity for a large number of hidden nodes in the hidden layer. The results show that it has constant training time which is closer towards the minimal training time and very far from worst-case training time of the DF-ELM algorithm that has been shown efficient in the recent literature.

Applying Neural-Network-Based Machine Learning to Additive Manufacturing: Current Applications, Challenges, and Future Perspectives

Abstract Additive manufacturing (AM), also known as three-dimensional printing, is gaining increasing attention from academia and industry due to the unique advantages it has in comparison with traditional subtractive manufacturing. However, AM processing parameters are difficult to tune, since they can exert a huge impact on the printed microstructure and on the performance of the subsequent products. It is a difficult task to build a process–structure–property–performance (PSPP) relationship for AM using traditional numerical and analytical models. Today, the machine learning (ML) method has been demonstrated to be a valid way to perform complex pattern recognition and regression analysis without an explicit need to construct and solve the underlying physical models. Among ML algorithms, the neural network (NN) is the most widely used model due to the large dataset that is currently available, strong computational power, and sophisticated algorithm architecture. This paper overviews the progress of applying the NN algorithm to several aspects of the AM whole chain, including model design, in situ monitoring, and quality evaluation. Current challenges in applying NNs to AM and potential solutions for these problems are then outlined. Finally, future trends are proposed in order to provide an overall discussion of this interdisciplinary area.

Anomaly Detection in ERP Systems Using AI and Machine Learning

ERP systems are critical in the administration of corporate activities and address huge volumes of transactional and operation information. However, since ERP systems combine many operations of an organization into one system it is prone to a possibility of developing an anomaly that can come from an erroneous data input or even instances of hacking hence causing operational insecurity and loss. This work seeks to understand how AI (Artificial Intelligence) and Machine Leaning (ML) can be used to determine abnormalities in ERP systems. Conventional methods of anomaly detection do not allow for detailed recognition and handling of complex patterns; thus, AI and ML are suitable for dynamic systems. The paper discusses different forms of anomalous situations in ERP systems and examines the potential of different learning techniques in increasing the effectiveness of anomaly identification. The framework that is presented provides for the incorporation of ML relied anomaly recognition in ERP systems to optimize operational efficiency as well as error identification in real time.

ReStoCNet: Residual Stochastic Binary Convolutional Spiking Neural Network for Memory-Efficient Neuromorphic Computing.

In this work, we propose ReStoCNet, a residual stochastic multilayer convolutional Spiking Neural Network (SNN) composed of binary kernels, to reduce the synaptic memory footprint and enhance the computational efficiency of SNNs for complex pattern recognition tasks. ReStoCNet consists of an input layer followed by stacked convolutional layers for hierarchical input feature extraction, pooling layers for dimensionality reduction, and fully-connected layer for inference. In addition, we introduce residual connections between the stacked convolutional layers to improve the hierarchical feature learning capability of deep SNNs. We propose Spike Timing Dependent Plasticity (STDP) based probabilistic learning algorithm, referred to as Hybrid-STDP (HB-STDP), incorporating Hebbian and anti-Hebbian learning mechanisms, to train the binary kernels forming ReStoCNet in a layer-wise unsupervised manner. We demonstrate the efficacy of ReStoCNet and the presented HB-STDP based unsupervised training methodology on the MNIST and CIFAR-10 datasets. We show that residual connections enable the deeper convolutional layers to self-learn useful high-level input features and mitigate the accuracy loss observed in deep SNNs devoid of residual connections. The proposed ReStoCNet offers >20 × kernel memory compression compared to full-precision (32-bit) SNN while yielding high enough classification accuracy on the chosen pattern recognition tasks.

Application of data mining approach to identify drug subclasses based on solubility and permeability.

Solubility and permeability are recognized as key parameters governing drug intestinal absorption and represent the basis for biopharmaceutics drug classification. The Biopharmaceutics Classification System (BCS) is widely accepted and adopted by regulatory agencies. However, currently established low/high permeability and solubility boundaries are the subject of the ongoing scientific discussion. The aim of the present study was to apply data mining analysis on the selected drugs data set in order to develop a human permeability predictive model based on selected molecular descriptors, and to perform data clustering and classification to identify drug subclasses with respect to dose/solubility ratio (D/S) and effective permeability (Peff ). The Peff values predicted for 30 model drugs for which experimental human permeability data are not available were in good agreement with the reported fraction of drug absorbed. The results of clustering and classification analysis indicate the predominant influence of Peff over D/S. Two Peff cut-off values (1×10-4 and 2.7×10-4 cm/s) have been identified indicating the existence of an intermediate group of drugs with moderate permeability. Advanced computational analysis employed in the present study enabled the recognition of complex relationships and patterns within physicochemical and biopharmaceutical properties associated with drug bioperformance.

Perspectives in Radiomics for Personalized Medicine and Theranostics.

Radiomics handles imaging biomarker from high-throughput feature extraction through complex pattern recognition that is difficult for human to process. Recent medical paradigms are rapidly changing to personalized medicine, including molecular targeted therapy, immunotherapy, and theranostics, and the importance of biomarkers for these is growing day by day. Even though biopsy continues to gold standard for tumor assessment in personalized medicine, imaging is expected to complement biopsy because it allows whole tumor evaluation, whole body evaluation, and non-invasive and repetitive evaluation. Radiomics is known as a useful method to get imaging biomarkers related to intratumor heterogeneity in molecular targeted therapy as well as one-size-fits-all therapy. It is also expected to be useful in new paradigms such as immunotherapy and somatostatin receptor (SSTR) or prostate-specific membrane antigen (PSMA)-targeted theranostics. Radiomics research should move to multimodality (CT, MR, PET, etc.), multicenter, and prospective studies from current single modality, single institution, and retrospective studies. Image-quality harmonization, intertumor heterogeneity, and integrative analysis of information from different scales are thought to be important keywords in future radiomics research. It is clear that radiomics will play an important role in personalized medicine.

Clustering Information Distance by Nested-Lattice Compression

Clustering of mixed dataset is one of the key issues in complex system architecture decomposition and many other pattern recognition applications. Recent development of machine learning techniques, especially deep neural network models significantly boost clustering performance, whereas theoretical analysis explaining why it works and how to further improve the performance is limited in literature. From the information theoretical point of view, information distance based on Kolmogorov complexity captures non-feature similarities and works as an absolute information measure. Thus, it is valuable to model the clustering process of mixed discrete and continuous datasets with this generalized similarity measure, and investigate the practical implementation of the information distance, compression distance, to make it work within the machine learning-enabled clustering schemes. In this paper, a generative adversarial network (GAN)-based deep clustering scheme is modified to compare the similarities of the original data in order to reduce the deviation of projected features from real data characteristics. The compression distance is extended to be computable for continuous data with the assistance of the Wyner-Ziv setup, where theoretical limit is given via asymptotically optimal nested lattices. Flatness-based analysis is carried out with distance upper bound derived for non-vanishing flatness factor. Practical distance calculation is derived to incorporate with neural-network-based clustering. Simulation via practical lattice codes illustrates the consistency of the design and shows its potential effectiveness in clustering applications.

Artificial optic-neural synapse for colored and color-mixed pattern recognition

The priority of synaptic device researches has been given to prove the device potential for the emulation of synaptic dynamics and not to functionalize further synaptic devices for more complex learning. Here, we demonstrate an optic-neural synaptic device by implementing synaptic and optical-sensing functions together on h-BN/WSe2 heterostructure. This device mimics the colored and color-mixed pattern recognition capabilities of the human vision system when arranged in an optic-neural network. Our synaptic device demonstrates a close to linear weight update trajectory while providing a large number of stable conduction states with less than 1% variation per state. The device operates with low voltage spikes of 0.3 V and consumes only 66 fJ per spike. This consequently facilitates the demonstration of accurate and energy efficient colored and color-mixed pattern recognition. The work will be an important step toward neural networks that comprise neural sensing and training functions for more complex pattern recognition.

On the von Neumann entropy of graphs

AbstractThe von Neumann entropy of a graph is a spectral complexity measure that has recently found applications in complex networks analysis and pattern recognition. Two variants of the von Neumann entropy exist based on the graph Laplacian and normalized graph Laplacian, respectively. Due to its computational complexity, previous works have proposed to approximate the von Neumann entropy, effectively reducing it to the computation of simple node degree statistics. Unfortunately, a number of issues surrounding the von Neumann entropy remain unsolved to date, including the interpretation of this spectral measure in terms of structural patterns, understanding the relation between its two variants and evaluating the quality of the corresponding approximations. In this article, we aim to answer these questions by first analysing and comparing the quadratic approximations of the two variants and then performing an extensive set of experiments on both synthetic and real-world graphs. We find that (1) the two entropies lead to the emergence of similar structures, but with some significant differences; (2) the correlation between them ranges from weakly positive to strongly negative, depending on the topology of the underlying graph; (3) the quadratic approximations fail to capture the presence of non-trivial structural patterns that seem to influence the value of the exact entropies; and (4) the quality of the approximations, as well as which variant of the von Neumann entropy is better approximated, depends on the topology of the underlying graph.