3D Image Representation Research Articles

Image representation of vibration signals and its application on fault diagnosis of rotating machinery with convolutional neural network: A benchmark study

Intelligent fault diagnosis methods on the basis of two-dimensional (2D) image representation of vibration signals (IRVS) and the convolutional neural network (CNN) have been extensively applied in rotating machinery. However, as a large number of IRVS methods are being used, their performance has not been fairly and comprehensively evaluated, which leads to difficulties for researchers to make choices. To address this issue, this paper conducted a benchmark study aimed at comparing the comprehensive performance of different IRVS methods, including computation time, fault diagnosis accuracy, and noise resistance performance. Firstly, the IRVS methods and the fault diagnosis method combining IRVS and CNN were summarized. Then, the general process of the IRVS-CNN method was proposed. Finally, 17 types of IRVS methods were selected, and the performance of the IRVS method was compared based on three datasets and two classic CNN models. The results indicate that the computing time taken to generate 2D images using unthresholded recurrence plot (UTRP), and frequency-domain unthresholded recurrence plot (FDUTRP) methods is relatively longer. The vast majority of frequency-domain methods and time-frequency domain methods can achieve or approach a high accuracy rate in all experiments, demonstrating their outstanding performance. Frequency-domain color vibration image (FDCVI), FDUTRP, and continuous wavelet transform (CWT) have good noise resistance. These results provide a reference for future researchers in selecting IRVS methods.

WALDATA: Wavelet transform based adversarial learning for the detection of anomalous trading activities

Detecting manipulative activities in stock market trading poses a significant challenge due to the complex temporal correlations inherent to the dynamically changing stock price data. This challenge is further exacerbated by the limited availability of labelled anomalous trading data instances. Stock price manipulations, which consist of infrequent anomalies in stock price trading data, are challenging to capture due to their sporadic occurrence and dynamically evolving nature. This scarcity and inherent complexity significantly complicate the creation of labelled datasets hence hinders the development of robust detection of different stock price manipulation schemes through supervised learning methods. Overcoming these challenges is crucial for enhancing our understanding of market dynamics and implementing robust market surveillance systems. To address these challenges, we introduce a novel stock price manipulation detection approach called WALDATA (Wavelet Transform based Adversarial Learning for the Detection of Anomalous Trading Activities). We leverage the Wavelet Transform (WT) to decompose non-stationary stock price time series into informative features and capture multi-scale dynamics within the data. We encode stock price data by transforming it into scalogram images through the Continuous Wavelet Transform, effectively converting stock price time series data into a 2D image representation. Subsequently, we employ a Generative Adversarial Network (GAN) architecture, originally applied to computer vision, to learn the underlying distribution of normal trading behaviour from the encoded images. We then train the discriminator as an anomaly detector for identifying manipulative trading activities in the stock market. The efficacy of WALDATA is rigorously evaluated on diverse real-world stock datasets using 1-level tick data from the LOBSTER project and the experimental results demonstrate the significant performance of our approach achieving an average AUC of 0.99 while maintaining low false alarm rates across various market conditions. These findings not only validate the effectiveness of the proposed WALDATA approach in accurately identifying stock price manipulations but also provide investors and regulators alike with valuable insights for the development of advanced market surveillance systems. This research demonstrates the promising potential of combining wavelet-based feature extraction and stock price time series to image representation with generative adversarial learning frameworks for anomaly detection in financial time series data. The successful implementation of WALDATA contributes to the development of advanced market surveillance systems and paves the way for further advancements in market surveillance, contributing towards a more efficient and robust financial system and a fair market environment.

Holographic pyramid as an instructional resource for inserting 3D images in the teaching of optical isomerism

The New Digital Information and Communication Technologies and their applications in the teaching of chemistry have been highlighted as a contributor to the promotion of abstract concepts. In search of innovations that can contribute to the teaching of Optical Isomerism, we investigated the collaboration of the teaching of this concept provided in a didactic sequence with a CTS focus and with the support of a pedagogical didactic resource, the holographic pyramid for the representation of 3D images. The research was qualitative in nature and the target audience consisted of 35 high school students from a state school located in Vera Cruz do Oeste, Paraná. The students took part in two lessons that made up a didactic sequence addressing "Optical Isomerism and Medicines", and based on the data collected, we analyzed the construction of knowledge and the establishment of the relationships necessary for learning. The methods of analysis were based on the procedures of Discursive Textual Analysis, categorization was adapted and carried out a priori using the deductive method with support from Bloom's Digital Taxonomy and the materials resulting from the application of the research were interpreted using the Double Coding Theory. We observed that some students contemplated the three levels of cognitive processing established by Paivio (1971): representational, referential and associative. However, for other students, we analyzed the need for different stimuli so that they could establish concrete connections. We suggest the Double Coding Theory for the analysis of teaching based on 3D images, contemplating the abstract concepts of Optical Isomerism and highlighting the importance of digital technology applied to chemistry teaching methods.

Global Feature Integration for Machinery Fault Diagnosis: The ViTransformer Model on Time-Frequency Representation

Traditional fault diagnosis in rotating machinery often relies on one-dimensional (1D) time-series data, which oftentimes may obscure spatial relationships and dynamic interactions between different machine components and operational states. This limitation can lead to incomplete or inaccurate fault detection, as critical interactions that occur across multiple dimensions are not represented. Moreover, methods that utilize visual plot images for diagnosis can introduce variations such as line width, plot resolution, and overlapping points or curves, which further complicate the analysis and obscure relevant data. To address these challenges, this study employs Time-Frequency Representations (TFRs) to transform vibration signals into two-dimensional (2D) images that encapsulate rich temporal and frequency information, thereby overcoming the shortcomings of both 1D data and conventional plot images. Further enhancing this approach, the Vision Transformer (ViT) model is integrated, utilizing its self-attention mechanism to effectively discern and emphasize globally significant features within these images. Validated on an in-house gearbox dataset, our methodology demonstrates significant improvements in diagnostic accuracy and efficiency, offering a robust, comprehensive tool for machinery fault analysis that leverages the advantages of detailed 2D image representation and advanced neural network technology.

Classification of rib fracture types from postmortem computed tomography images using deep learning.

Human or time resources can sometimes fall short in medical image diagnostics, and analyzing images in full detail can be a challenging task. With recent advances in artificial intelligence, an increasing number of systems have been developed to assist clinicians in their work. In this study, the objective was to train a model that can distinguish between various fracture types on different levels of hierarchical taxonomy and detect them on 2D-image representations of volumetric postmortem computed tomography (PMCT) data. We used a deep learning model based on the ResNet50 architecture that was pretrained on ImageNet data, and we used transfer learning to fine-tune it to our specific task. We trained our model to distinguish between "displaced," "nondisplaced," "ad latus," "ad longitudinem cum contractione," and "ad longitudinem cum distractione" fractures. Radiographs with no fractures were correctly predicted in 95-99% of cases. Nondisplaced fractures were correctly predicted in 80-86% of cases. Displaced fractures of the "ad latus" type were correctly predicted in 17-18% of cases. The other two displaced types of fractures, "ad longitudinem cum contractione" and "ad longitudinem cum distractione," were correctly predicted in 70-75% and 64-75% of cases, respectively. The model achieved the best performance when the level of hierarchical taxonomy was high, while it had more difficulties when the level of hierarchical taxonomy was lower. Overall, deep learning techniques constitute a reliable solution for forensic pathologists and medical practitioners seeking to reduce workload.

Brain activity underlying visual search in depth when viewing volumetric multiplanar images

The study investigated the cortical activity associated with 3D and 2D image perception on a volumetric multiplanar display by analyzing event-related potentials (ERPs) and power spectral density (PSD). In this study, we used a volumetric multiplanar display to present visual targets, and the brain signals were recorded via an EEG amplifier and analyzed using the EEGLAB toolbox on MATLAB. The study found no significant differences in amplitude between the 3D and 2D conditions across five occipital and parietal electrodes. However, there was a significant difference in latency of the P3 component on the Pz electrode. The analysis of PSD showed no significant differences between the two conditions, although there was a slightly higher alpha and beta activity observed in the 2D visualization. The study concluded that 3D image representation on a volumetric multiplanar display has no more sensory or cognitive load on the human brain than 2D representation, and that depth perception on a multiplanar display requires less brain activity.

Robust fault diagnosis of quayside container crane gearbox based on 2D image representation in frequency domain and CNN

To accurately diagnose the quayside container crane (QCC) gearbox faults, this article proposes a method that combines the frequency-domain Markov transformation field (FDMTF) and multi-branch residual convolutional neural network (MBRCNN). Firstly, the gearbox vibration signal is converted into the frequency domain to reveal the components and amplitude of signals stably and concisely. Then, the one-dimensional frequency signal is encoded into the two-dimensional image by the Markov transformation field to capture the dynamic characteristics of signals. Thirdly, the MBRCNN network is constructed, which can extract multi-scale features and alleviate the problems caused by the deep network structure. Finally, the FDMTF image is fed into the constructed MBRCNN model for pattern recognition. The effectiveness of the proposed FDMTF–MBRCNN method is verified by two case studies. In Case 1, the diagnosis results of a benchmark dataset achieve 100% accuracy, better than seven state-of-the-art methods published in recent 3 years. In Case 2, the diagnosis results of the dataset collected from a 1:4 scaled test rig achieve 98.85% accuracy, better than eleven encoding methods and four convolutional neural network methods. It also can obtain a recognition accuracy of more than 94% under the conditions of small sample, different network hyper-parameters, or variable loads, which verifies its robustness. These case studies show that the FDMTF–MBRCNN method is expected to be applied to the actual fault diagnosis of QCC gearboxes.

Non-stationary neural signal to image conversion framework for image-based deep learning algorithms.

This paper presents a time-efficient preprocessing framework that converts any given 1D physiological signal recordings into a 2D image representation for training image-based deep learning models. The non-stationary signal is rasterized into the 2D image using Bresenham's line algorithm with time complexity O(n). The robustness of the proposed approach is evaluated based on two publicly available datasets. This study classified three different neural spikes (multi-class) and EEG epileptic seizure and non-seizure (binary class) based on shapes using a modified 2D Convolution Neural Network (2D CNN). The multi-class dataset consists of artificially simulated neural recordings with different Signal-to-Noise Ratios (SNR). The 2D CNN architecture showed significant performance for all individual SNRs scores with (SNR/ACC): 0.5/99.69, 0.75/99.69, 1.0/99.49, 1.25/98.85, 1.5/97.43, 1.75/95.20 and 2.0/91.98. Additionally, the binary class dataset also achieved 97.52% accuracy by outperforming several others proposed algorithms. Likewise, this approach could be employed on other biomedical signals such as Electrocardiograph (EKG) and Electromyography (EMG).

Generating High-Resolution 3D Faces and Bodies Using VQ-VAE-2 with PixelSNAIL Networks on 2D Representations.

Modeling and representing 3D shapes of the human body and face is a prominent field due to its applications in the healthcare, clothes, and movie industry. In our work, we tackled the problem of 3D face and body synthesis by reducing 3D meshes to 2D image representations. We show that the face can naturally be modeled on a 2D grid. At the same time, for more challenging 3D body geometries, we proposed a novel non-bijective 3D-2D conversion method representing the 3D body mesh as a plurality of rendered projections on the 2D grid. Then, we trained a state-of-the-art vector-quantized variational autoencoder (VQ-VAE-2) to learn a latent representation of 2D images and fit a PixelSNAIL autoregressive model to sample novel synthetic meshes. We evaluated our method versus a classical one based on principal component analysis (PCA) by sampling from the empirical cumulative distribution of the PCA scores. We used the empirical distributions of two commonly used metrics, specificity and diversity, to quantitatively demonstrate that the synthetic faces generated with our method are statistically closer to real faces when compared with the PCA ones. Our experiment on the 3D body geometry requires further research to match the test set statistics but shows promising results.

An innovative deep anomaly detection of building energy consumption using energy time-series images

Deep anomaly detection (DAD) is essential in optimizing building energy management. Nonetheless, most existing works concerning this field consider unsupervised learning and involve the analysis of sensor readings through a one-dimensional (1D) energy time series, which limits the options for detecting anomalies within the building’s energy consumption. To the best of the authors’ knowledge, this paper presents the first study that explores using two-dimensional (2D) image representations as features of a supervised deep transfer learning (DTL) approach. Specifically, using 2D image representations allows taking advantage of any underlying data within the feature set, providing more possibilities to encode data and detect pertinent features which may not be considered in standard 1D time-series. Furthermore, the effects of using CNN activations as machine learning (ML) model features are also investigated to combine the advantages of both techniques. Additionally, the concept of layer and hyperparameter variation for the CNN model is also studied, with the objective of reducing the overall time computation and resource requirements of the proposed system. Hence, this makes our approach a candidate for edge-based applications. As per the conducted experiments, the top methodology rests at the F1-scores of 93.63% and 99.89% under simulated and real-world energy datasets, respectively. This involves using grayscale 2D image representations that combine CNN activations extracted from AlexNet and GoogleNet pre-trained models as features to a linear support vector machine (SVM) classifier. Finally, the comparison analysis with the state-of-the-art has shown the superiority of the proposed method in various assessment criteria.

A Multimodal Fusion Approach for Human Activity Recognition

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a "dynamic" image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

An inception V3 approach for malware classification using machine learning and transfer learning

Malware instances have been extremely used for illegitimate purposes, and new variants of malware are observed every day. Machine learning in network security is one of the prime areas of research today because of its performance and has shown tremendous growth in the last decade. In this paper, we formulate the malware signature as a 2D image representation and leverage deep learning approaches to characterize the signature of malware contained in BIG15 dataset across nine classes. The current research compares the performance of various machine learning and deep learning technologies towards malware classification such as Logistic Regression (LR), Artificial Neural Network (ANN), Convolutional Neural Network (CNN), transfer learning on CNN and Long Short Term Memory (LSTM). The transfer learning approach using InceptionV3 resulted in a good performance with respect to the compared models like LSTM with a classification accuracy of 98.76% on the test dataset and 99.6% on the train dataset.

Brain2Pix: Fully convolutional naturalistic video frame reconstruction from brain activity.

Reconstructing complex and dynamic visual perception from brain activity remains a major challenge in machine learning applications to neuroscience. Here, we present a new method for reconstructing naturalistic images and videos from very large single-participant functional magnetic resonance imaging data that leverages the recent success of image-to-image transformation networks. This is achieved by exploiting spatial information obtained from retinotopic mappings across the visual system. More specifically, we first determine what position each voxel in a particular region of interest would represent in the visual field based on its corresponding receptive field location. Then, the 2D image representation of the brain activity on the visual field is passed to a fully convolutional image-to-image network trained to recover the original stimuli using VGG feature loss with an adversarial regularizer. In our experiments, we show that our method offers a significant improvement over existing video reconstruction techniques.

Low-rank 2D local discriminant graph embedding for robust image feature extraction

As a popular feature extraction algorithm, the 2D local preserving projections (2DLPP) algorithm has been successfully applied in many fields. Using 2D image representation, the 2DLPP algorithm preserves the manifold attributes and retains the local information of high-dimensional space data. However, the 2DLPP algorithm may encounter some problems in real-world applications, such as a lack of discriminatory ability, singularity problems, and sensitivity to occlusion and noise in data. Therefore, this paper introduces low-rank into the 2DLPP algorithm and proposes a new feature extraction algorithm, which is the low-rank two-dimensional local discriminant graph embedding (LR-2DLDGE), to solve these problems. To improve the LR-2DLDGE algorithm robustness, we fuse the discriminant information in graph embedding and the low-rank properties of the data. The algorithm has three advantages: First, the algorithm uses a graph embedding (GE) framework to maintain the local neighbourhood discrimination information between data. Second, the LR-2DLDGE algorithm ensures that the data points are as independent as possible from different classes in the feature space. Finally, the algorithm uses the L1-norm as a constraint and reduces the influence of noise and corruption through low-rank learning. The theoretical computational complexity and convergence of the algorithm are explicated and proved. Extensive experimental results on three occluded and noisy image datasets confirm the effectively and robustness of LR-2DLDGE, respectively.

Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations.

We present Neural Feature Fusion Fields (N3F), a method that improves dense 2D image feature extractors when the latter are applied to the analysis of multiple images reconstructible as a 3D scene. Given an image feature extractor, for example pre-trained using self-supervision, N3F uses it as a teacher to learn a student network defined in 3D space. The 3D student network is similar to a neural radiance field that distills said features and can be trained with the usual differentiable rendering machinery. As a consequence, N3F is readily applicable to most neural rendering formulations, including vanilla NeRF and its extensions to complex dynamic scenes. We show that our method not only enables semantic understanding in the context of scene-specific neural fields without the use of manual labels,but also consistently improves over the self-supervised 2D baselines. This is demonstrated by considering various tasks, such as 2D object retrieval, 3D segmentation, and scene editing, in diverse sequences, including long egocentric videos in the EPIC-KITCHENS benchmark. Project page: https://www.robots.ox.ac.uk/~vadim/n3f/.

Representasi 3D Jaringan Epidermis dan Stomata Daun Beberapa Jenis Tumbuhan Suku Apocynaceae serta Sumbangannya pada Pembelajaran Biologi SMA

This study aims to produce a 3D image representation of leaf epidermis tissue in several Apocynaceae plants as enrichment material for high school biology material. The method used is a descriptive method by observing the leaf epidermis tissue in several Apocynaceae plants, namely: Adenium obesum, Catharanthus roseus, Cerbera manghas, Nerium oleander, Plumeria alba and Wrightia antidysenterica. The representation of 3D images is done using Paint-3D software and arranged in the form of a booklet. The results showed that the representation of the structure of the epidermis and stomata in the form of 3D images displayed varying characteristics. The results of the feasibility test show that the booklet has an average score of 93 and is included in the very valid category. In conclusion, the 3D image representation of the epidermal tissue with Paint-3D software can display the characteristics of the tissue structure more clearly and the booklets that are arranged are suitable for use as teaching materials.
 Keywords: Apocynaceae, Epidermis, 3D Representation, Stomata

Affinity-Point Graph Convolutional Network for 3D Point Cloud Analysis

Efficient learning of 3D shape representation from point cloud is one of the biggest requirements in 3D computer vision. In recent years, convolutional neural networks have achieved great success in 2D image representation learning. However, unlike images that have a Euclidean structure, 3D point clouds are irregular since the neighbors of each node are inconsistent. Many studies have tried to develop various convolutional graph neural networks to overcome this problem and to achieve great results. Nevertheless, these studies simply took the centroid point and its corresponding neighbors as the graph structure, thus ignoring the structural information. In this paper, an Affinity-Point Graph Convolutional Network (AP-GCN) is proposed to learn the graph structure for each reference point. In this method, the affinity between points is first defined using the feature of each point feature. Then, a graph with affinity information is built. After that, the edge-conditioned convolution is performed between the graph vertices and edges to obtain stronger neighborhood information. Finally, the learned information is used for recognition and segmentation tasks. Comprehensive experiments demonstrate that AP-GCN learned much more reasonable features and achieved significant improvements in 3D computer vision tasks such as object classification and segmentation.

Learning geometry-aware joint latent space for simultaneous multimodal shape generation

The real-world objects in our physical environment present with diverse information and multimodal features, including 3D shapes (geometry and topology) and 2D images (appearance and semantics), etc. How to effectively represent and correlate them in a unified way is still very challenging due to different modalities and representations. In this paper, we present a novel method to learn a unified and effective latent space for a joint representation and simultaneous generation of 3D point clouds and 2D images. We propose a new geometry-aware autoencoder for 3D shapes with a full-resolution shape feature extractor and a multi-resolution geometric feature extractor at different scales, which can enhance the geometric variability and scalability of the latent representation. Then, the proposed mixer, i.e., a joint latent space, can synergically integrate and complement the encoded features from 3D geometry and 2D contents through our intermodality feature mapping and intramodality feature consistency design. It is noted that our joint latent space can simultaneously generate multimodal representations and correlations with high-quality, high-fidelity, and high cross-modality similarity, which the traditional single-modal methods cannot handle. The extensive experiments demonstrate that our approach outperforms the state-of-the-art methods in shape auto-encoding as well as simultaneous multimodal (SMM) shape and color image generation and interpolation, etc. Furthermore, our joint-learning of 2D and 3D facets of a shape for the novel SMM semantic-aware generation task can enhance the capability of the corresponding single-modality and single-tasking to the next level.

Molecular persistent spectral image (Mol-PSI) representation for machine learning models in drug design.

Artificial intelligence (AI)-based drug design has great promise to fundamentally change the landscape of the pharmaceutical industry. Even though there are great progress from handcrafted feature-based machine learning models, 3D convolutional neural networks (CNNs) and graph neural networks, effective and efficient representations that characterize the structural, physical, chemical and biological properties of molecular structures and interactions remain to be a great challenge. Here, we propose an equal-sized molecular 2D image representation, known as the molecular persistent spectral image (Mol-PSI), and combine it with CNN model for AI-based drug design. Mol-PSI provides a unique one-to-one image representation for molecular structures and interactions. In general, deep models are empowered to achieve better performance with systematically organized representations in image format. A well-designed parallel CNN architecture for adapting Mol-PSIs is developed for protein-ligand binding affinity prediction. Our results, for the three most commonly used databases, including PDBbind-v2007, PDBbind-v2013 and PDBbind-v2016, are better than all traditional machine learning models, as far as we know. Our Mol-PSI model provides a powerful molecular representation that can be widely used in AI-based drug design and molecular data analysis.

Research on fan vibration fault diagnosis based on image recognition

The conventional methods for vibration fault detection and diagnosis relies on feature extraction from the waveforms of the vibration signals. This article exploits the scope of image recognition application for the detection and diagnosis of fan vibration faults. In this paper, a novel image recognition technique is proposed for vibration-based fault diagnosis using the spectrum images of the vibration signals. 1D vibration signal spectrum is initially computed using Fast Fourier Transform (FFT) and the FFT frequencies are adjusted such that it captures a vibration spectrum diagram as 2D image representation. FFT based vibration analysis is done and the image recognition concept is utilized for feature extraction and a machine learning classification module is used for fault analysis and diagnosis. Effective feature generation is done using Principal Component Analysis (PCA) by removing the redundancy from the feature vectors and machine learning classifiers are used to obtain improved image recognition and classification performance. Artificial Neural Network (ANN) classifier yields better performance in terms of various performance parameters and percentage improvement in terms of accuracy for ANN classification methods over Support Vector Machine (SVM), k-Nearest Neighbours (kNN) and Random Forest Ensemble (RFE) methods are 10.01 %, 4.51 % and 2.01 % respectively. Comparative scenarios are considered in this work for fan vibration fault detection as well as diagnosis based on the image features for various realistic vibration fault conditions. Effectiveness of the proposed image recognition-based technique is compared with the state-of-the-art methods, justifying its outperformance for fan fault detection and diagnosis using the combination of spectrum adjustment, PCA and ANN classification method.