Image-based Representation Research Articles

Exploring the Impact of Image-Based Audio Representations in Classification Tasks Using Vision Transformers and Explainable AI Techniques

An important hurdle in medical diagnostics is the high-quality and interpretable classification of audio signals. In this study, we present an image-based representation of infant crying audio files to predict abnormal infant cries using a vision transformer and also show significant improvements in the performance and interpretability of this computer-aided tool. The use of advanced feature extraction techniques such as Gammatone Frequency Cepstral Coefficients (GFCCs) resulted in a classification accuracy of 96.33%. For other features (spectrogram and mel-spectrogram), the performance was very similar, with an accuracy of 93.17% for the spectrogram and 94.83% accuracy for the mel-spectrogram. We used our vision transformer (ViT) model, which is less complex but more effective than the proposed audio spectrogram transformer (AST). We incorporated explainable AI (XAI) techniques such as Layer-wise Relevance Propagation (LRP), Local Interpretable Model-agnostic Explanations (LIME), and attention mechanisms to ensure transparency and reliability in decision-making, which helped us understand the why of model predictions. The accuracy of detection was higher than previously reported and the results were easy to interpret, demonstrating that this work can potentially serve as a new benchmark for audio classification tasks, especially in medical diagnostics, and providing better prospects for an imminent future of trustworthy AI-based healthcare solutions.

Advancing cardiac diagnostics: Exceptional accuracy in abnormal ECG signal classification with cascading deep learning and explainability analysis

Arrhythmias, cardiac rhythm disorders, demand precise diagnosis for effective treatment planning, emphasizing the crucial role of electrocardiogram (ECG) signal interpretation. While deep learning excels in ECG classification, its interpretability remains a challenge. Addressing this issue, our research introduces an innovative approach employing image-based ECG representations and cascading deep neural networks (CDNNs) to enhance arrhythmia detection precision. Initiating with the transformation of 12-lead ECG time-series data, particularly focusing on Lead II, into image-based representations using relative positioning matrices (RPM), this novel conversion captures spatial and temporal information concurrently, enriching the depth of our data analysis. Subsequently, CDNNs play a pivotal role in feature extraction, designed to automatically extract impactful features from image-based ECG representations, providing profound insights into cardiac electrical activity. Extracted features serve as inputs for the subsequent deep neural network for classification. Experimental results on the Shaoxing–Chapman ECG database, encompassing over 10,000 recordings with an 80/10/10 split, 80/20 split, 60/40 split, and 10-fold cross-validation, showcase a remarkable 100% accuracy in distinguishing between seven and four different arrhythmia classes. Incorporating SHapley Additive exPlanations (SHAP) values, our methodology offers comprehensive insights into the model’s decision-making, ensuring interpretability. Additionally, Gradient-weighted Class Activation Mapping (Grad-CAM) aids feature visualization, elucidating key features driving predictions. By amalgamating image-based representation, CDNNs, and interpretability techniques, our research not only attains exceptional classification performance but also boosts model transparency and trustworthiness. These findings hold promise for advancing arrhythmia diagnosis and enhancing interpretability in deep learning-based medical applications.

Developing a Data Model for an Omnidirectional Image-Based Multi-Scale Representation of Space

Abstract. One of the major challenges that existing spatial data is facing is the fragmentation of its representation of indoor and outdoor space. As studies in the use of omnidirectional images in representing space and providing Location-based Services (LBS) has been increasing, the representation of the different scales of space, both in indoors and outdoors, has yet to be addressed. This study aims to develop a data model for generating a multi-scale image-based representation of space using omnidirectional images based spatial relationships. This paper identifies the different scales of space that are represented in spatial data and extends previous approaches of using omnidirectional images in providing indoor LBS towards representing the other scales of space, particularly in outdoor space. Using a sample data, we present an experimental implementation to demonstrate the potential of the proposed data model. Results show that apart from the realistic visualization that image data provides, basic spatial functions can be performed on the image data constructed based on the proposed data model.

Bolt load looseness detection for slip-critical blind bolt based on wavelet analysis and deep learning

Bolt preload is a crucial factor in high-strength bolt applications, particularly for the Slip-critical blind bolts (SCBBs). This study presents a novel nondestructive approach that leverages ultrasonic echo waves to accurately detect the state of bolt looseness, addressing a significant research gap in the field of blind bolt looseness detection. The proposed technique is uniquely suitable for blind bolted connections as it only necessitates access to a single side of the connection. Nine types of SCBBs were tested and obtained approximately 4000 ultrasonic echo signals with varying degrees of looseness. These signals have been transformed into image-based representations using wavelet analysis, and deep learning techniques were used to classify and predict the looseness level accurately. The performance of various damage assessment criteria, CNN models, and dataset sizes were evaluated and compared. The proposed method was validated by classifying 400 datasets with a validated accuracy of 97.30%.

Deep learning of movement behavior profiles and their association with markers of cardiometabolic health

BackgroundTraditionally, existing studies assessing the health associations of accelerometer-measured movement behaviors have been performed with few averaged values, mainly representing the duration of physical activities and sedentary behaviors. Such averaged values cannot naturally capture the complex interplay between the duration, timing, and patterns of accumulation of movement behaviors, that altogether may be codependently related to health outcomes in adults. In this study, we introduce a novel approach to visually represent recorded movement behaviors as images using original accelerometer outputs. Subsequently, we utilize these images for cluster analysis employing deep convolutional autoencoders.MethodsOur method involves converting minute-by-minute accelerometer outputs (activity counts) into a 2D image format, capturing the entire spectrum of movement behaviors performed by each participant. By utilizing convolutional autoencoders, we enable the learning of these image-based representations. Subsequently, we apply the K-means algorithm to cluster these learned representations. We used data from 1812 adult (20–65 years) participants in the National Health and Nutrition Examination Survey (NHANES, 2003–2006 cycles) study who worn a hip-worn accelerometer for 7 seven consecutive days and provided valid accelerometer data.ResultsDeep convolutional autoencoders were able to learn the image representation, encompassing the entire spectrum of movement behaviors. The images were encoded into 32 latent variables, and cluster analysis based on these learned representations for the movement behavior images resulted in the identification of four distinct movement behavior profiles characterized by varying levels, timing, and patterns of accumulation of movement behaviors. After adjusting for potential covariates, the movement behavior profile characterized as “Early-morning movers” and the profile characterized as “Highest activity” both had lower levels of insulin (P < 0.01 for both), triglycerides (P < 0.05 and P < 0.01, respectively), HOMA-IR (P < 0.01 for both), and plasma glucose (P < 0.05 and P < 0.1, respectively) compared to the “Lowest activity” profile. No significant differences were observed for the “Least sedentary movers” profile compared to the “Lowest activity” profile.ConclusionsDeep learning of movement behavior profiles revealed that, in addition to duration and patterns of movement behaviors, the timing of physical activity may also be crucial for gaining additional health benefits.

IMTRAC: Image-Based Modeling for Transformative Traffic Control via Foundation Models

This paper introduces the Image-based Modeling for Transformative Traffic Control (IMTRAC) framework, which revolutionizes urban traffic management by integrating image-based representations, foundation models, and dynamic signaling. It transitions from textual to visual data analysis, transforming complex traffic scenes into quantifiable insights. The image-based representation is then analyzed by foundation models to infer refined and effective traffic control policies. Our experimental results show marked improvements in traffic flow and reductions in waiting times, highlighting IMTRAC’s ability to outperform traditional control methods. The proposed IMTRAC promises a significant leap towards smarter, efficient urban traffic systems, leveraging the synergy of advanced imaging techniques and multi-modal foundation models to enhance control decisions.

Time Series from Sentinel-2 for Organic Durum Wheat Yield Prediction Using Functional Data Analysis and Deep Learning

Yield prediction is a crucial activity in scheduling agronomic operations and in informing the management and financial decisions of a wide range of stakeholders of the organic durum wheat supply chain. This research aims to develop a yield forecasting system by combining vegetation index time-series data from Sentinel-2 L2A time-series data, field-measured yields, and deep learning techniques. Remotely sensed data over a season could be, in general, noisy and characterized by a variable density due to weather conditions. This problem was mitigated using Functional Principal Component Analysis (FPCA). We obtained a functional representation of acquired data, and starting from this, we tried to apply deep learning to predict the crop yield. We used a Convolutional Neural Network (CNN) approach, starting from images that embed temporal and spectral dimensions. This representation does not require one to a priori select a vegetation index that, typically, is task-dependent. The results have been also compared with classical approaches as Partial Least Squares (PLS) on the main reference vegetation indexes such as the Normalized Difference Vegetation Index (NDVI) and Normalized Difference Red Edge index (NDRE), considering both in-season and end-season scenarios. The obtained results show that the image-based representation of multi-spectral time series could be an effective method to estimate the yield, also, in the middle stage of cropping with R2 values greater than 0.83. The developed model could be used to estimate yield the neighbor fields characterized by similar setups in terms of the crop, variety, soil, and, of course, management.

Automated algorithm selection using meta-learning and pre-trained deep convolution neural networks

Automated machine learning (AutoML), which explores the automation of various machine learning tasks, has seen significant progress in recent years. One of the fundamental tasks in this field is the selection of effective algorithms for a given dataset. This task is particularly challenging for tabular datasets where, unlike images or text, one cannot use pre-trained embeddings from other datasets. In this study, we present AutoIRAD, a novel meta-learning and vision-based approach for automatic classification algorithm selection for tabular datasets. Our approach is the first to generate image-based representations of entire tabular datasets, enabling us to model the features and interactions of all the dataset samples. Our image-based representation has two significant advantages. First, training our model across multiple datasets enables it to simultaneously learn from diverse domains and create more generalized and effective dataset representations. This representation can also be effectively applied to previously unseen datasets. Secondly, using images enables us to leverage large pre-trained networks (e.g., VGG). By tuning existing networks rather than training new ones, we can both speed up the training of our model and reduce the needed amounts of training data. An extensive evaluation of 150 datasets shows that AutoIRAD achieves results comparable to the leading state-of-the-art AutoML solution but at a fraction of their computation time.

Ecological experimentations

The goal of this paper is to highlight the ecological aspect in the works of fi lmmaker Rose Lowder. Her fi lms are shot outdoors, in nature, and aim to modify visual perception and image-based representation, foregrounding the materiality of the fi lmic medium. Her fi lmography, characterized by its use of unique shooting and editing techniques, is remarkable for its calculation regarding creative possibilities. The parameters of investigation outlined in this study are the fi lmic procedures that stress the aesthetic and ecological qualities of this artist’s work

Towards efficient image-based representation of tabular data

Towards efficient image-based representation of tabular data

Resource-Aware Time Series Imaging Classification for Wireless Link Layer Anomalies.

The number of end devices that use the last-mile wireless connectivity is dramatically increasing with the rise of smart infrastructures and requires reliable functioning to support smooth and efficient business processes. To efficiently manage such massive wireless networks, more advanced and accurate network monitoring and malfunction detection solutions are required. In this article, we perform a first-time analysis of image-based representation techniques for wireless anomaly detection using recurrence plots (RPs) and Gramian angular fields and propose a new deep learning architecture enabling accurate anomaly detection. We elaborate on the design considerations for developing a resource-aware architecture and propose a new model using time series to image transformation using RPs. We show that the proposed model: 1) outperforms the one based on Gramian angular fields by up to 14% points; 2) outperforms classical ML models using dynamic time warping by up to 24% points; 3) outperforms or performs on par with mainstream architectures, such as AlexNet and VGG11 while having <10x their weights and up to ≈ 8% of their computational complexity; and d) outperforms the state of the art in the respective application area by up to 55% points. Finally, we also explain on randomly chosen examples how the classifier takes decisions.

Score-based generative modeling for de novo protein design.

The generation of de novo protein structures with predefined functions and properties remains a challenging problem in protein design. Diffusion models, also known as score-based generative models (SGMs), have recently exhibited astounding empirical performance in image synthesis. Here we use image-based representations of protein structure to develop ProteinSGM, a score-based generative model that produces realistic de novo proteins. Through unconditional generation, we show that ProteinSGM can generate native-like protein structures, surpassing the performance of previously reported generative models. We experimentally validate some de novo designs and observe secondary structure compositions consistent with generated backbones. Finally, we apply conditional generation to de novo protein design by formulating it as an image inpainting problem, allowing precise and modular design of protein structure.

Bioelectrical impedance analysis of thermal-induced cutaneous nociception

Objective:People perceive different levels of pain at the same stimulus intensity, hence overburdening objective nociception assessment. In the context of recently developed pain monitors, this work presents the analysis of the complex bioimpedance measured during cold-induced cutaneous nociception. Using the Anspec-PRO device, this analysis results in a more complex characterization of the acquired signal (i.e., 3-dimensional or image-based representations), rather than evaluations of one/multiple single-dimension signals (i.e., time-based index). Methods:The experimental protocol consists of thermal testing the nociception response of 13 participants, in terms of impedance changes. Several study cases are performed to evaluate the bioimpedance variability over one day, day to day, and at different stimulus intensities. Results:The measured bioimpedance fluctuations in time were validated against the self-reported pain level of the test subjects. Moreover, the bioimpedance, acting as the frequency response of cold-induced nociception, demonstrated the pain memory effect and the influence on the impedance of the liquid/fat in the body. Conclusion:The impedance analysis can delineate cold-induced nociception effects from nominal states, the impedance spectroscopy proving the potential to monitor the evoked pain. Significance:The implications of this work rely on advancing the state of art for pain detection by using time–frequency analysis and image-based information. Moreover, in closed-loop anesthesia control, the nociception surrogate monitoring provides direct feedback for analgesia and an input–output model to opioids.

Touchscreen gestures as images. A transfer learning approach for soft biometric traits recognition

Mobile devices are nowadays ubiquitous. They are equipped with a variety of sensors, each designed for capturing specific signals which can be exploited to acquire discriminating user’s traits, thus allowing to recognize, for instance, the authorized users. In this regard, we focus on capturing soft biometric traits from smartphones. Soft biometric information extracted from a human body (e.g., gender and age) is ancillary information proved to improve the performance of biometric authentication systems, and it has drawn a great deal of attention for its applications in healthcare, smart spaces and digital world as a whole. This paper presents an approach to gender and age-group recognition, namely TGSB, leveraging a transfer learning strategy applied to state-of-the-art Convolutional Neural Networks (CNNs) fed with image-based representations of touch gestures performed by users on mobile devices. We perform experiments considering, one at a time, touch gestures of the same kind, and combinations thereof, with intermediate and late fusion learning strategies. Experiments prove that TGSB is a promising approach, with up to 94% accuracy for the gender recognition and up to 99% for the age-group recognition. We highlight the most useful touch gesture for gender and age-group recognition, that is Scroll with 81% and 96% accuracy, respectively. We show that combining multiple touch gestures (intermediate fusion) with a joint latent subspace learning mechanism in CNNs improves the TGSB performance, up to 99% accuracy when considering a combination of two Scroll. Compared to previous works, TGSB exhibits much better performance in both gender and age-group recognition.

A saliency mapping approach to understanding the visual impact of wind and solar infrastructure in amenity landscapes

ABSTRACT Shifts from fossil fuels toward renewable energy (RE) introduce profound changes to landscapes, including visual impacts that are often investigated during environmental and social impact assessment. Moreover, RE transitions are among many visual changes happening in rural areas that are increasingly serving amenity functions and becoming destinations for wide ranges of users. This diversity introduces complexities during infrastructure siting discussions. Emerging grape and wine production landscapes in Canada serve amenity and production purposes, and this study was designed to understand the impacts of RE development using case studies of solar panels and wind turbines in two vineyard landscapes in Ontario (ON) and British Columbia (BC). We applied novel mixed methods, including content analysis and saliency-based visual impact analysis, to textual and image-based representations posted on Instagram of those vineyards. In this case, the addition of low-density RE infrastructures did not seem to disturb the vineyard experience. The technique presented can be applied in a wide range of infrastructure siting contexts, both before and after construction, but requires additional research and calibration.

A Transfer Learning Framework for Predictive Energy-Related Scenarios in Smart Buildings

Human activities and city routines follow patterns. Transfer learning can help achieve scalable solutions toward the realization of smart cities accounting for similarities between regions, domains, and activities. In this study, we propose a transfer learning-based framework for smart buildings to test this hypothesis in energy-related problems. Our framework has two major components: the network creation and the transferable predictive model. In order to create the network that groups buildings sharing characteristics, we evaluated two strategies: a novel clustering algorithm for mixed data, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</i> -prod, and clustering the image-based representation of time series. Then, a combination of long short term memory and convolutional neural network was trained on the centroids of the clusters for energy consumption prediction. The coefficient of variation of the root mean squared error (CVRMSE) of the predictions in such clusters vary between 3.85% and 58.85%. The obtained parameters were transferred to the rest of the buildings for predictive purposes, finding accurate results in buildings with little data. Our framework deals with insufficient training data since parameters from scenarios with more sensors can be received. It also carries out state-of-the-art performance on three datasets from different sources having in total 533 rooms/buildings and two energy efficiency domains: consumption prediction reducing the CVRMSE in a 21.6%, and air conditioning usage prediction moving from a 4.18% to a 0.28% CVRMSE. Our framework extracts more knowledge from available IoT deployments, so that smartness could be spread between environments at a fewer cost given that less individual effort will be needed.

Robust Event-Based Vision Model Estimation by Dispersion Minimisation.

We propose a novel Dispersion Minimisation framework for event-based vision model estimation, with applications to optical flow and high-speed motion estimation. The framework extends previous event-based motion compensation algorithms by avoiding computing an optimisation score based on an explicit image-based representation, which provides three main benefits: i) The framework can be extended to perform incremental estimation, i.e., on an event-by-event basis. ii) Besides purely visual transformations in 2D, the framework can readily use additional information, e.g., by augmenting the events with depth, to estimate the parameters of motion models in higher dimensional spaces. iii) The optimisation complexity only depends on the number of events. We achieve this by modelling the event alignment according to candidate parameters and minimising the resultant dispersion, which is computed by a family of suitable entropy-based measures. Data whitening is also proposed as a simple and effective pre-processing step to make the framework's accuracy performance more robust, as well as other event-based motion-compensation methods. The framework is evaluated on several challenging motion estimation problems, including 6-DOF transformation, rotational motion, and optical flow estimation, achieving state-of-the-art performance.

Spatio-Temporal Image Representation and Deep-Learning-Based Decision Framework for Automated Vehicles

Driving maneuver decision-making is critical to the development and mass deployment of automated vehicles (AVs). The prevailing approaches are stuck with either specific optimization objectives or separate maneuver planning. Inspired by the inherent driving mechanism of human beings, we develop a comprehensive deep learning scheme that extracts and establishes informative image-based representation of dynamic traffic flows and then learns driving strategies from collected driving datasets based on convolutional neural networks (CNNs). Our scheme can effectively capture comprehensive and dynamic features of traffic flows surrounding the ego car and construct dynamic motion images through processing the spatio-temporal signals of all neighbor cars. These constructed virtual images cover all informative dynamic states of the neighbor cars, located in the maneuver-critical area, defined as the motion-sensitive area (MSA). Then, sequential spatio-temporal images together with labeled driving behaviors are fed into our proposed CNN model. The network can extract underlying motion patterns and learn proper driving behaviors, including both the lateral maneuver and longitudinal speed. We demonstrate the effectiveness of the proposed scheme in a typical highway scenario. Results suggest that the proposed scheme merits further investigation to promote the launch of AVs.

FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality.

Virtual Reality (VR) is becoming ubiquitous with the rise of consumer displays and commercial VR platforms. Such displays require low latency and high quality rendering of synthetic imagery with reduced compute overheads. Recent advances in neural rendering showed promise of unlocking new possibilities in 3D computer graphics via image-based representations of virtual or physical environments. Specifically, the neural radiance fields (NeRF) demonstrated that photo-realistic quality and continuous view changes of 3D scenes can be achieved without loss of view-dependent effects. While NeRF can significantly benefit rendering for VR applications, it faces unique challenges posed by high field-of-view, high resolution, and stereoscopic/egocentric viewing, typically causing low quality and high latency of the rendered images. In VR, this not only harms the interaction experience but may also cause sickness. To tackle these problems toward six-degrees-of-freedom, egocentric, and stereo NeRF in VR, we present the first gaze-contingent 3D neural representation and view synthesis method. We incorporate the human psychophysics of visual- and stereo-acuity into an egocentric neural representation of 3D scenery. We then jointly optimize the latency/performance and visual quality while mutually bridging human perception and neural scene synthesis to achieve perceptually high-quality immersive interaction. We conducted both objective analysis and subjective studies to evaluate the effectiveness of our approach. We find that our method significantly reduces latency (up to 99% time reduction compared with NeRF) without loss of high-fidelity rendering (perceptually identical to full-resolution ground truth). The presented approach may serve as the first step toward future VR/AR systems that capture, teleport, and visualize remote environments in real-time.

Prediction of Alzheimer's Disease by a Novel Image-Based Representation of Gene Expression.

Early intervention can delay the progress of Alzheimer’s Disease (AD), but currently, there are no effective prediction tools. The goal of this study is to generate a reliable artificial intelligence (AI) model capable of detecting the high risk of AD, based on gene expression arrays from blood samples. To that end, a novel image-formation method is proposed to transform single-dimension gene expressions into a discriminative 2-dimensional (2D) image to use convolutional neural networks (CNNs) for classification. Three publicly available datasets were pooled, and a total of 11,618 common genes’ expression values were obtained. The genes were then categorized for their discriminating power using the Fisher distance (AD vs. control (CTL)) and mapped to a 2D image by linear discriminant analysis (LDA). Then, a six-layer CNN model with 292,493 parameters were used for classification. An accuracy of 0.842 and an area under curve (AUC) of 0.875 were achieved for the AD vs. CTL classification. The proposed method obtained higher accuracy and AUC compared with other reported methods. The conversion to 2D in CNN offers a unique advantage for improving accuracy and can be easily transferred to the clinic to drastically improve AD (or any disease) early detection.