Spectrogram Images Research Articles

MSIC: Malware Spectrogram Image Classification

The heavy reliance on digital technology, by individuals and organizations, has reshaped the traditional economy into a digital economy. In response, cybercriminals' attention has shifted dramatically from showing off skills and conducting individual attacks into high sophisticated attacks with financial gain as the goal. This, inevitably, poses a challenge to the cybersecurity community as they strive to find solutions to preserve the confidentiality, availability and integrity of the individual users' and corporates' private data and services. Cybercriminals mainly deploy malware to achieve their goals, which could be in the form of ransomware, botnets, etc. The use of encryption, packing and polymorphism techniques makes it harder to detect the malware files, especially when these are created in great numbers every day. In this paper, a novel framework, named Malware Spectrogram Image Classification (MSIC), is proposed. It employs spectrogram images in conjunction with the convolution neural network to classify a malware file to its corresponding family and to differentiate it from a benign file. Further, this research shares with the research community two privately collected labeled malicious and benign datasets. The evaluation of MSIC showed its effectiveness to be 91.6% F-measure and 92.8% accuracy in classifying malware files to their corresponding families, in comparison to, respectively, 90.6% and 92.3% results produced by the grayscale image classification approach. Likewise, in classifying files as malicious or benign, MSIC scored 96% F-measure and accuracy results compared to 95.5% with the grayscale solution. Also, MSIC required less computational time in converting and resizing the files than the grayscale framework.

A New Deep CNN Model for Environmental Sound Classification

Cognitive prediction in the complicated and active environments is of great importance role in artificial learning. Classification accuracy of sound events has a robust relation with the feature extraction. In this paper, deep features are used in the environmental sound classification (ESC) problem. The deep features are extracted by using the fully connected layers of a newly developed Convolutional Neural Networks (CNN) model, which is trained in the end-to-end fashion with the spectrogram images. The feature vector is constituted with concatenating of the fully connected layers of the proposed CNN model. For testing the performance of the proposed method, the feature set is conveyed as input to the random subspaces K Nearest Neighbor (KNN) ensembles classifier. The experimental studies, which are carried out on the DCASE-2017 ASC and the UrbanSound8K datasets, show that the proposed CNN model achieves classification accuracies 96.23% and 86.70%, respectively.

Prediction of Inter-Personal Trust and Team Familiarity From Speech: A Double Transfer Learning Approach

Speech classification is one of the most convenient objective measures of internal state exhibited during a problem-solving task that requires verbal communication. This study investigates the hypothesis of speech acoustic characteristics being indicative of trust between team members and team members’ familiarity with each other. Speech recordings from 27 dyadic teams (26 males and 28 females) were made during a distributed threat perception task, determining safe points along a route through the town to be visited by a VIP. Before the threat detection mission, 26 team members knew each other, and the remaining 28 had no prior knowledge of their partners. Two levels (Low Trust and High Trust) of two trust constructs, TTP (Trust, Trustworthiness, Propensity to trust), and RIS (Reliance Intentions Scale), were estimated based on numerical responses to pre- and post-mission surveys. Speech recordings of individual speakers were divided into 1-second intervals and converted into RGB images of amplitude spectrograms. The images were classified using a pre-trained convolutional neural network ResNet-18 fine-tuned to recognize either the trust level or familiarity. In the baseline classification scenario, the speech was classified using a single transfer learning into Low/High-trust categories separately for RIS and TTP constructs before and after the mission yielding an average classification accuracy of 82%-86%. Single transfer learning classification into Know/Unknown-partners categories led to 85% accuracy. Application of double transfer learning, i.e., first tuning the ResNet-18 on Know/Unknown labels and then on Low/High-trust, increased the trust classification accuracy up to 89%. When tuning the ResNet-18 on Low/High-trust and then on Known/Unknown labels, the accuracy of partner familiarity recognition was also increased up to 89%. These results support the hypothesis of speech acoustics being indicative of trust and familiarity between team members and show that by adding prior related knowledge to the model, more efficient learning can be achieved without increasing the training data size.

Drill Fault Diagnosis Based on the Scalogram and Mel Spectrogram of Sound Signals Using Artificial Intelligence

In industry, the ability to detect damage or abnormal functioning in machinery is very important. However, manual detection of machine fault sound is economically inefficient and labor-intensive. Hence, automatic machine fault detection (MFD) plays an important role in reducing operating and personnel costs compared to manual machine fault detection. This research aims to develop a drill fault detection system using state-of-the-art artificial intelligence techniques. Many researchers have applied the traditional approach design for an MFD system, including handcrafted feature extraction of the raw sound signal, feature selection, and conventional classification. However, drill sound fault detection based on conventional machine learning methods using the raw sound signal in the time domain faces a number of challenges. For example, it can be difficult to extract and select good features to input in a classifier, and the accuracy of fault detection may not be sufficient to meet industrial requirements. Hence, we propose a method that uses deep learning architecture to extract rich features from the image representation of sound signals combined with machine learning classifiers to classify drill fault sounds of drilling machines. The proposed methods are trained and evaluated using the real sound dataset provided by the factory. The experiment results show a good classification accuracy of 80.25 percent when using Mel spectrogram and scalogram images. The results promise significant potential for using in the fault diagnosis support system based on the sounds of drilling machines.

Neural Network Music Genre Classification

Music genre classification utilizing neural networks (NNs) has achieved some limited success in recent years. Differences in song libraries, machine learning techniques, input formats, and types of NNs implemented have all had varying levels of success. This article reviews some of the machine learning techniques utilized in this area. It also presents research work on music genre classification. The research uses images of spectrograms generated from timeslices of songs as the input into an NN to classify the songs into their respective musical genres.

A Novel Approach for Classification of Speech Emotions Based on Deep and Acoustic Features

The problem of recognition and classification of emotions in speech is one of the most prominent research topics, that has gained popularity, in human-computer interaction in the last decades. Having recognized the feelings or emotions in human conversations might have a deep impact on understanding a human’s physical and psychological situation. This study proposes a novel hybrid architecture based on acoustic and deep features to increase the classification accuracy in the problem of speech emotion recognition. The proposed method consists of feature extraction, feature selection and classification stages. At first, acoustic features such as Root Mean Square energy (RMS), Mel-Frequency Cepstral Coefficients (MFCC) and Zero-crossing Rate are obtained from voice records. Subsequently, spectrogram images of the original sound signals are given as input to the pre-trained deep network architecture, which is VGG16, ResNet18, ResNet50, ResNet101, SqueezeNet and DenseNet201 and deep features are extracted. Thereafter, a hybrid feature vector is created by combining acoustic and deep features. Also, the ReliefF algorithm is used to select more efficient features from the hybrid feature vector. Finally, in order for the completion of the classification task, Support vector machine (SVM) is used. Experiments are made using three popular datasets used in the literature so as to evaluate the effect of various techniques. These datasets are Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), Berlin (EMO-DB) and Interactive Emotional Dyadic Motion Capture (IEMOCAP). As a consequence, we reach to 79.41%, 90.21% and 85.37% accuracy rates for RAVDESS, EMO-DB, and IEMOCAP datasets, respectively. The Final results obtained in experiments, clearly, show that the proposed technique might be utilized to accomplish the task of speech emotion recognition efficiently. Moreover, when our technique is compared with those of methods used in the context, it is obvious that our method outperforms others in terms of classification accuracy rates.

Exposing Speech Resampling Manipulation by Local Texture Analysis on Spectrogram Images

Speech tampering may be aided by the resampling operation. It is significant for speech forensics to effectively detect the resampling; however, there are few studies on speech resampling detection. The purpose of this paper was therefore to provide a new training ideal to detect speech resampling. After resampling, the speech signal changes regularly in the time–frequency domain. In this paper, we theoretically analyzed the corresponding relationship between time domain and frequency domain of the resampled speech. Compared with the original speech, the bandwidth of resampled speech was stretched or compressed. First, the spectrogram was generated by short-time Fourier transform (STFT) from the speech. Then, the local binary pattern (LBP) operator was applied to model the statistical changes in the spectrogram and the LBP histogram was calculated as discriminative features. Finally, a support vector machine (SVM) was applied to classify the developed features to identify whether the speech had undergone the resampling operation. The experimental results show that the proposed method has superior detection performance in different resampling scenarios than some existing methods, and the proposed features are very robust against the commonly used compression post-processing operation. This highlights the promising potential of the proposed method as a speech resampling detection tool in practical forensics applications.

Convolutional neural networks based efficient approach for classification of lung diseases

Treatment of lung diseases, which are the third most common cause of death in the world, is of great importance in the medical field. Many studies using lung sounds recorded with stethoscope have been conducted in the literature in order to diagnose the lung diseases with artificial intelligence-compatible devices and to assist the experts in their diagnosis. In this paper, ICBHI 2017 database which includes different sample frequencies, noise and background sounds was used for the classification of lung sounds. The lung sound signals were initially converted to spectrogram images by using time–frequency method. The short time Fourier transform (STFT) method was considered as time–frequency transformation. Two deep learning based approaches were used for lung sound classification. In the first approach, a pre-trained deep convolutional neural networks (CNN) model was used for feature extraction and a support vector machine (SVM) classifier was used in classification of the lung sounds. In the second approach, the pre-trained deep CNN model was fine-tuned (transfer learning) via spectrogram images for lung sound classification. The accuracies of the proposed methods were tested by using the ten-fold cross validation. The accuracies for the first and second proposed methods were 65.5% and 63.09%, respectively. The obtained accuracies were then compared with some of the existing results and it was seen that obtained scores were better than the other results.

Automatic bird song and syllable segmentation with an open-source deep-learning object detection method – a case study in the Collared Flycatcher (Ficedula albicollis)

Abstract The bioacoustic analyses of animal sounds result in an enormous amount of digitized acoustic data, and we need effective automatic processing to extract the information content of the recordings. Our research focuses on the song of Collared Flycatcher (Ficedula albicollis) and we are interested in the evolution of acoustic signals. During the last 20 years, we obtained hundreds of hours of recordings of bird songs collected in natural environment, and there is a permanent need for the automatic process of recordings. In this study, we chose an open-source, deep-learning image detection system to (1) find the species-specific songs of the Collared Flycatcher on the recordings and (2) to detect the small, discrete elements so-called syllables within the song. For these tasks, we first transformed the acoustic data into spectrogram images, then we trained two deep-learning models separately on our manually segmented database. The resulted models detect the songs with an intersection of union higher than 0.8 and the syllables higher than 0.7. This technique anticipates an order of magnitude less human effort in the acoustic processing than the manual method used before. Thanks to the new technique, we are able to address new biological questions that need large amount of acoustic data.

Detection of Cardiac Arrhythmia using Machine Learning Algorithms

Cardiac Arrhythmia is a type of condition a human being suffers from abnormal heart rhythm. This is experienced due to the malfunctioning of electrical impulses that coordinate the heartbeat. When this happens the heartbeats slow/ fast more precisely irregularly. The rhythm of the heart is controlled by a major node called the sinus node which is present at the top of the heart, triggers the electrical pulses which make the heart to beat and pumping of blood to the body. Some of the symptoms of Cardiac Arrhythmia are fainting, unconsciousness, shortness of breath, unexpected functioning of the heart. It leads to death in minutes if medical attention is not provided. To diagnose this doctor, require to study the heart recordings evaluate heartbeats from different parts of the body accurately. It takes a lot of time to evaluate so based on the research work contributed in this field we try to propose a different approach to the same. In this paper, we compare different machine learning techniques and algorithms proposed by different authors and understand the advantages and disadvantages of the system and to bring a new system in place of the existing system where all have used the same ECG recordings from the same database of MIT-BIH. With the initial research work done by us we found out that the use of Phonocardiogram Recordings (PCG) provides more fidelity and accurate compared to ECG recordings. With the initial stage of work, we take the PCG recordings dataset and convert it to a spectrogram image and apply a convolutional neural network to predict the normal or abnormal heartbeat.

Active contour-based detection of estuarine dolphin whistles in spectrogram images

An algorithm for detecting tonal vocalizations from estuarine dolphin (Sotalia guianensis) specimens without interference of a human operator was developed. The raw audio data collected from a passive monitoring sensor in the Cananéia underwater soundscape was converted to spectrogram images. Detection is a four-step method: first, ridge maps are obtained from the spectrogram images; second, a probabilistic Hough transform algorithm is applied to detect ridges similar to thick line segments, referred to as line-like, which are adjusted to the geometry of the whistles in the images via an active contour algorithm; third, feature vectors are built from the geometry of each detected whistle, with 9 descriptive features; and fourth, the detections are fed to a random forest classifier to parse out mistakes by the detection process. We developed a system for classifying the characteristic patterns detected as Sotalia guianensis whistles or random empty detections. We obtained accuracy of 0.977 and F1-score of 0.981.

An Automated method for the analysis of bearing vibration based on spectrogram pattern matching

As a mean for non-intrusive inspection of bearing systems, the scope of predicting their condition from the acoustic vibrations liberated during their operation, utilizing signal processing methods, has been of extensive research, over decades. Vibration being highly non-stationary, time domain as well as spectral features cannot characterize its behavior. Even though spectrogram is a time-frequency domain feature extraction technique, its interpretation is tedious and perhaps, subjective. In the proposed method, the spectrogram images of the normal vibration data is compared with that of the contextual vibration, using Structural Similarity Index Metric (SSIM). It is hypothesized that the pattern similarity between the contextual spectrogram and the baseline is low when the bearing is faulty. The SSIM between the spectrogram image of normal bearing vibration data and the baseline is different from those between the baseline and vibration data corresponding to Inner Race Failure (IRF), Roller Element Defect (RED) and Outer Race Failure (ORF). Via the proposed method of spectrogram pattern matching based on SSIM, the subjectivity in the comparative interpretation of spectrogram is eliminated fully. The SSIM corresponding to the vibrations acquired from the normal and faulty bearings differ with a P value of 4.43693x 10-16. The technique can distinguish defective bearings with, 95.74% sensitivity, 96% accuracy and 100% specificity, without dismantling or open intervention

A new event detection method for noisy hydrophone data

In this paper, a new method for detecting events in noisy hydrophone data is developed. The method takes an image processing approach to the 1D hydrophone data by first converting it into a log-frequency spectrogram image (cepstrum). This image is then filtered by reconstructing it based on mutual information (MI) criteria of the dominant orientation map. The features of the reconstructed cepstrum are then enhanced using a combination of edge-tracking and noise smoothing. Binary feature classification on the processed cepstrum is performed using a least-squares support vector machine (LS-SVM). Compared to other methods, the proposed image-based event detection method exploits both the scale and the orientation information. The method showed that the detection performance in terms of binary classification sensitivity with more than 55% for rare events such as whale calls from noisy hydrophone recordings (a database size of over 160 h (1943 × 5 min) with a total of 39 events) from the NEPTUNE Canada project, with more than 99% specificity and 98% overall accuracy. With relatively low computational cost and high accuracy, the presented method is useful for automated long-term monitoring of a wide variety of marine mammals and human related activities from hydrophone data. The effectiveness of the proposed method is demonstrated over a large number of real noisy hydrophone recordings.

An Effective Hybrid Model for EEG-Based Drowsiness Detection

Early detection of driver drowsiness and the development of a functioning driver alertness system may support the prevention of numerous vehicular accidents worldwide. Wearable sensors and camera-based systems are generally employed in the driver drowsiness detection. Electroencephalogram (or EEG) is considered another effective option for the driver drowsiness detection. Various EEG-based drowsiness detection systems have been proposed to date. In this paper, EEG signals are also used for the detection of drowsiness, with the proposed method being composed of three main building blocks. Both raw EEG signals and their corresponding spectrograms are used in the proposed building blocks. In the first building block, while energy distribution and zero-crossing distribution features are calculated from the raw EEG signals, spectral entropy and instantaneous frequency features are extracted from the EEG spectrogram images. In the second building block, deep feature extraction is employed directly on the EEG spectrogram images using pre-trained AlexNet and VGGNet. In the third building block, the tunable Q-factor wavelet transform (TQWT) is used to decompose the EEG signals into related sub-bands. The spectrogram images of the obtained sub-bands and statistical features, such as mean and standard deviation of the sub-bands’ instantaneous frequencies, are then calculated. Each feature group from each building block is fed to a long-short term memory (LSTM) network for the purposes of classification. The obtained results from the LSTM networks are then fused with a majority voting layer. The MIT-BIH Polysomnographic database was used in the experimental works. The evaluation of the proposed method was carried out with ten-fold cross validation test and the average accuracy represented accordingly. The obtained average accuracy score was 94.31%. The obtained result was also compared with other results to be found in the literature. The comparison shows that the proposed method’s achievement was found to be better than the compared results.

A Novel Technique in Classifying Heart Diseases based on Electrocardiogram (ECG) Signals using Deep Learning and Spectrogram Image Analysis

A Novel Technique in Classifying Heart Diseases based on Electrocardiogram (ECG) Signals using Deep Learning and Spectrogram Image Analysis

Tracheal Sound Analysis Using a Deep Neural Network to Detect Sleep Apnea.

Portable devices for home sleep apnea testing are often limited by their inability to discriminate sleep/wake status, possibly resulting in underestimations. Tracheal sound (TS), which can be visualized as a spectrogram, carries information about apnea/hypopnea and sleep/wake status. We hypothesized that image analysis of all-night TS recordings by a deep neural network (DNN) would be capable of detecting breathing events and classifying sleep/wake status. The aim of this study is to develop a DNN-based system for sleep apnea testing and validate it using a large sampling of polysomnography (PSG) data. PSG examinations for the evaluation of sleep-disordered breathing (SDB) were performed for 1,852 patients: 1,548 PSG records were used to develop the system, and the remaining 304 records were used for validation. TS spectrogram images were obtained every 60 seconds and labeled with the PSG scoring results (breathing event and sleep/wake status), then introduced to DNN learning. Two different DNNs were trained for breathing status and sleep/wake status, respectively. A DNN with convolutional layers showed the best performance for discriminating breathing status. The same DNN structure was trained for sleep/wake discrimination. In the validation study, the DNN analysis was capable of discriminating the sleep/wake status with reasonable accuracy. The diagnostic sensitivity, specificity, and area under the receiver operating characteristic curves for diagnosis of SDB with apnea-hypopnea index of > 5, 15, and 30 were 0.98, 0.76, and 0.99; 0.97, 0.90, and 0.99; and 0.92, 0.94, and 0.98, respectively. The developed system using the TS DNN analysis has a good performance for SDB testing. Nakano H, Furukawa T, Tanigawa T. Tracheal sound analysis using a deep neural network to detect sleep apnea. J Clin Sleep Med. 2019;15(8): 1125-1133.

Towards the classification of heart sounds based on convolutional deep neural network.

Heart sound contains various important quantities that help early detection of heart diseases. Many methods have been proposed so far where various signal-processing techniques have been used on heart sounds for heart disease detection. In this paper, a methodology is introduced for heart disease detection based on heart sounds. The proposed method employs three successive stages, such as spectrogram generation, deep feature extraction, and classification. In the spectrogram generation stage, the heart sounds are converted to spectrogram images by using time-frequency transformation. The deep features are extracted from three different pre-trained convolutional neural network models such as AlexNet, VGG16, and VGG19. Support vector machine classifier is used in the third stage of the proposed method. The proposed method is evaluated on two datasets, which are taken from The Classifying Heart Sounds Challenge. The obtained results are compared with some of the existing methods. The comparisons show that the proposed method outperformed.

Event detection method comparison for distributed acoustic sensors using φ-OTDR

Abstract We investigate two kinds of event detection methods, support vector machine (SVM) and extreme learning machine (ELM), for distributed acoustic sensors (DAS) using phase-sensitive optical time domain reflectometer (φ-OTDR). We propose a heterodyne coherent φ-OTDR platform using an electric mixer, select spectrogram image feature (SIF) as a feature vector, and make comparison of the performance of 4 event classifiers. Theoretical and experimental analysis point out that an ELM classifier with RBF kernel is optimal in the detection of 5 sound events that has high identification accuracy, less training time, less testing time and strong noise immunity, which promises to be the first choice of event detection method in φ-OTDR systems in industrial applications.

Image Processing of Case Study of Solar Radio Burst Type III and Type IV Burst Using Sunpy

Solar radio burst at low-frequency lies in the fact that they originate in the same layers of the solar atmosphere in which geo-effective disturbance probably originate Type III bursts have consistently the fastest drift rates of bursts at metric wavelengths and was created by an electron beam through plasma emission. Meanwhile, type IV burst is broadband quasi-continuum features associated with the decay phase of solar flares. A geomagnetic storm is a major disturbance of Earth’s magnetosphere that occurs when there is a very efficient exchange of energy from the solar wind into the space environment surrounding Earth. The aim of this study is to find out whether type III bursts preserved energy to solar bursts type IV thus led to the formation of geomagnetic storm. The solar radio burst type III and IV were observed by CALLISTO Spectrometer. CALLISTO is an acronym from Compound Astronomical Low cost Low frequency Instrument for Spectroscopy and Transportable Observatory. The Sunpy by Python was used to analyze the spectrogram image by eliminate the unwanted signal or noise in the image which is in the presence of noise, the spectrum gives inaccurate reading of frequency and time. Although the study did not show much correlation between bursts type IV and type III in producing these storms, there are several possible explanations from this result. It may be the case that solar events such as solar flare, CMEs and active region ejected from the solar disc is not heading towards Earth and the energy has eventually disperses into the interplanetary medium.

Time\u2013Frequency Feature Fusion for Noise Robust Audio Event Classification

This paper explores the use of three different two-dimensional time–frequency features for audio event classification with deep neural network back-end classifiers. The evaluations use spectrogram, cochleogram and constant-Q transform-based images for classification of 50 classes of audio events in varying levels of acoustic background noise, revealing interesting performance patterns with respect to noise level, feature image type and classifier. Evidence is obtained that two well-performing features, the spectrogram and cochleogram, make use of information that is potentially complementary in the input features. Feature fusion is thus explored for each pair of features, as well as for all tested features. Results indicate that a fusion of spectrogram and cochleogram information is particularly beneficial, yielding an impressive 50-class accuracy of over 96% in 0 dB SNR and exceeding 99% accuracy in 10 dB SNR and above. Meanwhile, the cochleogram image feature is found to perform well in extreme noise cases of -,5 dB and -,10 dB SNR.