Deep Learning Paradigm Research Articles

Integrated deep learning paradigm for document-based sentiment analysis

An integrated deep learning paradigm for the analysis of document-based sentiments is presented in this article. Generally, sentiment analysis has enormous applications in the real world, particularly in e-commerce and/or cloud computing-oriented businesses. Integrated deep learning paradigms for document-based sentiment analysis seek to efficiently categorize the polarity of contextual sentiments into positive, negative, and neutral to aid organizations in making informed decisions. Nonetheless, the sparsity of text and disambiguation of natural languages make it relatively difficult for existing methods to provide precise identification and extraction when subjected to document-based data. As a result, this study introduces BERT-MultiLayered Convolutional Neural Network (B-MLCNN) as a computationally viable integrated deep learning paradigm. The B-MLCNN considers the overall textual review as a single document and classifies the available sentiments. First, the BERT pre-trained language model handles the feature vector representation and captures any global features. Further, the multi-layered convolutional neural network (MLCNN) with different kernel dimensions handles feature extraction. A softmax function produces classification results. The experimental setup with B-MLCNN based on IMDB movie reviews, 2002 movie reviews, 2004 movie reviews, and Amazon review datasets achieved accuracies of 95%, 88%, 95%, and 95% respectively, which promises to be efficient to deploy in practical applications.

Omnidirectional visual computing: Foundations, challenges, and applications

Omnidirectional images and videos are widespread due to the popularization of devices for capturing and visualization. Unlike pinhole-based imagery, omnidirectional media lie on the surface of a sphere, have a 360° × 180°field of view, and store the light intensities from an entire environment. Notably, applications involving immersive augmented, mixed, and virtual reality experiences benefit from the 360° content. Although defined on the spherical domain, omnidirectional images are often mapped to a (multi-)planar representation, which results in distorted images and degrades the performance of most traditional visual computing algorithms designed to work on the plane. This paper reviews the spherical camera model, the most common capturing devices, and popular (multi-)planar representations of omnidirectional media. It also approaches the main challenges of omnidirectional visual computing, focusing on the deep learning paradigm, and tackles four visual computing applications that strongly explore the potential of omnidirectional imagery.

Peer-to-Peer Federated Learning for COVID-19 Detection Using Transformers

The simultaneous advances in deep learning and the Internet of Things (IoT) have benefited distributed deep learning paradigms. Federated learning is one of the most promising frameworks, where a server works with local learners to train a global model. The intrinsic heterogeneity of IoT devices, or non-independent and identically distributed (Non-I.I.D.) data, combined with the unstable communication network environment, causes a bottleneck that slows convergence and degrades learning efficiency. Additionally, the majority of weight averaging-based model aggregation approaches raise questions about learning fairness. In this paper, we propose a peer-to-peer federated learning (P2PFL) framework based on Vision Transformers (ViT) models to help solve some of the above issues and classify COVID-19 vs. normal cases on Chest-X-Ray (CXR) images. Particularly, clients jointly iterate and aggregate the models in order to build a robust model. The experimental results demonstrate that the proposed approach is capable of significantly improving the performance of the model with an Area Under Curve (AUC) of 0.92 and 0.99 for hospital-1 and hospital-2, respectively.

Optical neural network via loose neuron array and functional learning

This research proposes a deep-learning paradigm, termed functional learning (FL), to physically train a loose neuron array, a group of non-handcrafted, non-differentiable, and loosely connected physical neurons whose connections and gradients are beyond explicit expression. The paradigm targets training non-differentiable hardware, and therefore solves many interdisciplinary challenges at once: the precise modeling and control of high-dimensional systems, the on-site calibration of multimodal hardware imperfectness, and the end-to-end training of non-differentiable and modeless physical neurons through implicit gradient propagation. It offers a methodology to build hardware without handcrafted design, strict fabrication, and precise assembling, thus forging paths for hardware design, chip manufacturing, physical neuron training, and system control. In addition, the functional learning paradigm is numerically and physically verified with an original light field neural network (LFNN). It realizes a programmable incoherent optical neural network, a well-known challenge that delivers light-speed, high-bandwidth, and power-efficient neural network inference via processing parallel visible light signals in the free space. As a promising supplement to existing power- and bandwidth-constrained digital neural networks, light field neural network has various potential applications: brain-inspired optical computation, high-bandwidth power-efficient neural network inference, and light-speed programmable lens/displays/detectors that operate in visible light.

Deep Pixel Restoration Loop Coding Network [Application Notes

This paper presents a solution to the video prediction problem based on the deep learning paradigm. Predicted frames generated by existing video prediction models are often blurry and have difficulty maintaining accuracy in multi-step prediction. To overcome these limitations, this paper presents a deep learning model, named the deep pixel restoration loop coding network (DPR-LC-Net), which employs the concept of predictive coding and adopts pixels in the real frames. While making a long-term prediction, it can generate clear prediction frames with few errors. The calculation process of DPR-LC-Net is multi-sequential: it completes calculations in the form of an approximate loop from top to bottom and then from left to right. After predicting subsequent steps, DPR-LC-Net both calculates errors and removes them from sequential prediction. Finally, the model includes a unique pixel restoration module that works efficiently on pixels in the preceding real frames to generate predicted frames, thereby improving the clarity of the predicted frames. Extensive experiments using four video datasets illustrate that the prediction performance of DPR-LC-Net is superior to that of the state-of-the-art models.

Deep Learning Paradigms for Existing and Imminent Lung Diseases Detection: A Review

Diagnosis of lung diseases like asthma, chronic obstructive pulmonary disease, tuberculosis, cancer, etc., by clinicians rely on images taken through various means like X-ray and MRI. Deep Learning (DL) paradigm has magnified growth in the medical image field in current years. With the advancement of DL, lung diseases in medical images can be efficiently identified and classified. For example, DL can detect lung cancer with an accuracy of 99.49% in supervised models and 95.3% in unsupervised models. The deep learning models can extract unattended features that can be effortlessly combined into the DL network architecture for better medical image examination of one or two lung diseases. In this review article, effective techniques are reviewed under the elementary DL models, viz. supervised, semi-supervised, and unsupervised Learning to represent the growth of DL in lung disease detection with lesser human intervention. Recent techniques are added to understand the paradigm shift and future research prospects. All three techniques used Computed Tomography (C.T.) images datasets till 2019, but after the pandemic period, chest radiographs (X-rays) datasets are more commonly used. X-rays help in the economically early detection of lung diseases that will save lives by providing early treatment. Each DL model focuses on identifying a few features of lung diseases. Researchers can explore the DL to automate the detection of more lung diseases through a standard system using datasets of X-ray images. Unsupervised DL has been extended from detection to prediction of lung diseases, which is a critical milestone to seek out the odds of lung sickness before it happens. Researchers can work on more prediction models identifying the severity stages of multiple lung diseases to reduce mortality rates and the associated cost. The review article aims to help researchers explore Deep Learning systems that can efficiently identify and predict lung diseases at enhanced accuracy.

Flock optimization induced deep learning for improved diabetes disease classification

AbstractDiabetic disease classification requires a precise understanding of the clinical inputs and their intensity as observed through different stages. Automated and machine‐centric classification requires validated data handling and non‐converging inputs. For improving the classification precision impacted due by complex computations, this article introduces an assimilated method incorporating flock optimization and conventional deep learning. Deep learning trains the classification system through the best‐fit solution generated by the flock optimization. The features from the input data are first identified for which an initial population is initiated. The identified features are classified based on their leap‐up behaviour; this behaviour is induced if the data feature modifies the actual representation. If the data feature shows up over‐fitting behaviour, then it is classified as abnormal and is discarded. Therefore the objective function is to identify the best‐fitting data feature from the maximum flock members showing similar leap‐up behaviour. This output is used for training the deep learning paradigm for classifying precision‐less and high features. The precision is determined using existing classified data that matches better the flock output. If the classified data is under less precision, then the leap‐up behaviours' objective is tuned to eliminate over‐fitting inputs. Therefore, the variable features are thwarted for preventing precision degradation for varying diabetics' clinical observed data. The introduced system maximize the recognition accuracy by 8.47% and minimize the complexity by 7.65%.

Drug repurposing for viral cancers: A paradigm of machine learning, deep learning, and virtual screening-based approaches.

Cancer management is major concern of health organizations and viral cancers account for approximately 15.4% of all known human cancers. Due to large number of patients, efficient treatments for viral cancers are needed. De novo drug discovery is time consuming and expensive process with high failure rate in clinical stages. To address this problem and provide treatments to patients suffering from viral cancers faster, drug repurposing emerges as an effective alternative which aims to find the other indications of the Food and Drug Administration approved drugs. Applied to viral cancers, drug repurposing studies following the niche have tried to find if already existing drugs could be used to treat viral cancers. Multiple drug repurposing approaches till date have been introduced with successful results in viral cancers and many drugs have been successfully repurposed various viral cancers. Here in this study, a critical review of viral cancer related databases, tools, and different machine learning, deep learning and virtual screening-based drug repurposing studies focusing on viral cancers is provided. Additionally, the mechanism of viral cancers is presented along with drug repurposing case study specific to each viral cancer. Finally, the limitations and challenges of various approaches along with possible solutions are provided.

Transformer Performance for Chemical Reactions: Analysis of Different Predictive and Evaluation Scenarios.

The prediction of chemical reaction pathways has been accelerated by the development of novel machine learning architectures based on the deep learning paradigm. In this context, deep neural networks initially designed for language translation have been used to accurately predict a wide range of chemical reactions. Among models suited for the task of language translation, the recently introduced molecular transformer reached impressive performance in terms of forward-synthesis and retrosynthesis predictions. In this study, we first present an analysis of the performance of transformer models for product, reactant, and reagent prediction tasks under different scenarios of data availability and data augmentation. We find that the impact of data augmentation depends on the prediction task and on the metric used to evaluate the model performance. Second, we probe the contribution of different combinations of input formats, tokenization schemes, and embedding strategies to model performance. We find that less stable input settings generally lead to better performance. Lastly, we validate the superiority of round-trip accuracy over simpler evaluation metrics, such as top-k accuracy, using a committee of human experts and show a strong agreement for predictions that pass the round-trip test. This demonstrates the usefulness of more elaborate metrics in complex predictive scenarios and highlights the limitations of direct comparisons to a predefined database, which may include a limited number of chemical reaction pathways.

A novel global solar exposure forecasting model based on air temperature: Designing a new multi-processing ensemble deep learning paradigm

The total quantity of solar energy falling on a horizontal plane surface is the global solar exposure (GSE, i.e., total solar energy). Precise forecasting of GSE is important in many fields such as renewable energy, agriculture, and public health, particularly by the limited hydro-meteorological time series information. This research aims to develop an advanced multi-processing deep learning (DL) paradigm to forecast weekly GSE based on maximum (Tmax) and minimum (Tmin) air temperatures as the drivers at Brisbane and Perth airport stations in eastern and western Australia during 2000 to 2022. The proposed model was comprised of an extra tree feature selection (FS) integrated with two novel decomposition techniques, namely time-varying filtering-based empirical mode decomposition (TVF-EMD), and empirical wavelet transform (EWT), and a powerful ensemble deep random vector functional link (ED-RVFL) approach. To validate the main model, the RVFL, bidirectional long-short term memory (Bi-LSTM), and bagged regression tree (Bagging) machine learning (ML) models were examined in hybrid and standalone counterpart frameworks. First, the extra-tree FS determined the significant lags of the predictors based on an importance benchmark criterion. Then, by applying the optimal gained lags to the feeding models, all of the original predictors were decomposed using TVF-EMD and EWT univariate feature extraction. The final forecast was computed by aggregating all the individual forecasts of the intrinsic mode functions (IMFs) and residual components. In addition, eight statistical indicators (including coefficient of correlation: R, root mean square error: RMSE, Kling-Gupta efficiency: KGE, index of agreement: IA, uncertainty coefficient with 95% confidence level: U95%, mean absolute percent error: MAPE, Nash-Sutcliffe efficiency: NSE, and mean absolute error: MAE) and several graphical methods were utilized to evaluate the performances of the models. The modeling results indicated that the ED-RVFL-TVF-EMD (R = 0.9665, RMSE = 1.9193 MJ/m2, and KGE = 0.9565 for the Perth airport station) and ED-RVFL-EWT (R = 0.9218, RMSE = 1.9708 MJ/m2, and KGE = 0.8552 for the Brisbane station) outperformed all other models, followed by RVFL, Bi-LSTM, and Bagging in a hybrid format. With the high predictive robustness, both decomposition-based frameworks can be useful for solving energy forecasting problems. The new modeling approach developed in this study can provide more precise forecasts for decision-makers to better address climate change, agriculture, and energy crises.

Deep Neural Network Modeling for CFD Simulations: Benchmarking the Fourier Neural Operator on the Lid-Driven Cavity Case

In this work we present the development, testing and comparison of three different physics-informed deep learning paradigms, namely the ConvLSTM, CNN-LSTM and a novel Fourier Neural Operator (FNO), for solving the partial differential equations of the RANS turbulence model. The 2D lid-driven cavity flow was chosen as our system of interest, and a dataset was generated using OpenFOAM. For this task, the models underwent hyperparameter optimization, prior to testing the effects of embedding physical information on performance. We used the mass conservation of the model solution, embedded as a term in our loss penalty, as our physical information. This approach has been shown to give physical coherence to the model results. Based on the performance, the ConvLSTM and FNO models were assessed in forecasting the flow for various combinations of input and output timestep sizes. The FNO model trained to forecast one timestep from one input timestep performed the best, with an RMSE for the overall x and y velocity components of 0.0060743 m·s−1.

Diagnosis of COVID-19 from Multimodal Imaging Data Using Optimized Deep Learning Techniques.

COVID-19 had a global impact, claiming many lives and disrupting healthcare systems even in many developed countries. Various mutations of the severe acute respiratory syndrome coronavirus-2, continue to be an impediment to early detection of this disease, which is vital for social well-being. Deep learning paradigm has been widely applied to investigate multimodal medical image data such as chest X-rays and CT scan images to aid in early detection and decision making about disease containment and treatment. Any method for reliable and accurate screening of COVID-19 infection would be beneficial for rapid detection as well as reducing direct virus exposure in healthcare professionals. Convolutional neural networks (CNN) have previously proven to be quite successful in the classification of medical images. A CNN is used in this study to suggest a deep learning classification method for detecting COVID-19 from chest X-ray images and CT scans. Samples from the Kaggle repository were collected to analyse model performance. Deep learning-based CNN models such as VGG-19, ResNet-50, Inception v3 and Xception models are optimized and compared by evaluating their accuracy after pre-processing the data. Because X-ray is a less expensive process than CT scan, chest X-ray images are considered to have a significant impact on COVID-19 screening. According to this work, chest X-rays outperform CT scans in terms of detection accuracy. The fine-tuned VGG-19 model detected COVID-19 with high accuracy-up to 94.17% for chest X-rays and 93% for CT scans. This work thereby concludes that VGG-19 was found to be the best suited model to detect COVID-19 and chest X-rays yield better accuracy than CT scans for the model.

An IoT aware nature inspired Multilayer Hybrid Dropout Deep-learning paradigm for waste image classification and management

AbstractIn this manuscript, the combination of IoT and Multilayer Hybrid Dropout Deep-learning Model for waste image categorization is proposed to categorize the wastes as bio waste and non-bio waste. The input captured images are pre-processed and remove noises in the captured images. Under this approach, a Nature inspired Multilayer Hybrid Dropout Deep-learning Model is proposed. Multilayer Hybrid Dropout Deep-learning Model is the consolidation of deep convolutional neural network and Dropout Extreme Learning Machine classifier. Here, deep convolutional neural network is used for feature extraction and Dropout Extreme Learning Machine classifier for categorizing the waste images. To improve the classification accurateness, Horse herd optimization algorithm is used to optimize the parameter of the Dropout Extreme Learning Machine classifier. The objective function is to maximize the accuracy by minimize the computational complexity. The simulation is executed in MATLAB. The proposed Multilayer Hybrid Dropout Deep-learning Model and Horse herd optimization algorithm attains higher accuracy 39.56% and 42.46%, higher Precision 48.74% and 34.56%, higher F-Score 32.5% and 45.34%, higher Sensitivity 24.45% and 34.23%, higher Specificity 31.43% and 21.45%, lower execution time 0.019(s) and 0.014(s) compared with existing waste management and classification using convolutional neural network with hyper parameter of random search optimization algorithm waste management and classification using clustering approach with Ant colony optimization algorithm. Finally, the proposed method categorizes the waste image accurately.

Human Regular Activities Recognition Using Convolutional Neural Network

Capturing commonly occurring behaviors is a tough issue in computer vision. A few of them are recreation, touring, leisure pursuits, and religious practice. A comprehensive effort has already been dedicated to this aspect to deal with this issue. In this work, we recreated a dataset with five categories, including household activities, farming, exercise, sports, and occupation, to identify human daily actions. This collection has 4328 colored images in total, among them 630 are set aside for testing, and 3698 for training. Deep learning and standard image-based strategies are being explored to address the issues. In this paper, we have designed a deep learning paradigm to classify the regular activities of human beings. To characterize people's daily chores, we use the CNN model, one of the greatest tools for visual identification. We also have chosen two already-trained VGG16 and ResNet50 models. When we compare our model with the existing techniques, the investigation's findings demonstrate that the suggested network has a better recognition accuracy of 91%. Additionally, we have observed that accuracy varies throughout different epochs, and after 25 epochs we got better stable results from our model. The reader may find this article instructive in grasping CNN models for various recognizing applications.

SGAE: Stacked Graph Autoencoder for Deep Clustering

Unsupervised clustering is a crucial issue in data mining and pattern recognition. Based on deep learning paradigms, deep clustering algorithms have been studied extensively and obtained superior performance in various applications. However, most of previous methods did not use helpful information from neighborhood relations to form group-separated space, and the feature embedding is usually distorted during the training process. To tackle the former limitation, we develop a graph convolution based unsupervised learning algorithm named Stacked Graph Autoencoder (SGAE). Specifically, SGAE utilizes the message passing mechanism to aggregate information from neighbors and obtain a meaningful and separated latent representation. Since the adjacency matrix is unavailable in clustering tasks, a graph construction approach with two pruning strategies is introduced to generate a transition matrix. To reduce the distortion caused by the multi-layered network training process, we further propose a topological structure preservation mechanism. It uses the constructed adjacency graph as supervised information, to maintain the relationship between nodes in the original space. Experiments on several popular benchmark datasets show that SGAE achieves significant improvements compared to unsupervised and semi-supervised deep clustering methods.

Forecasting Energy Consumption of a Public Building Using Transformer and Support Vector Regression

Most of the Artificial Intelligence (AI) models currently used in energy forecasting are traditional and deterministic. Recently, a novel deep learning paradigm, called ‘transformer’, has been developed, which adopts the mechanism of self-attention. Transformers are designed to better process and predict sequential data sets (i.e., historical time records) as well as to track any relationship in the sequential data. So far, a few transformer-based applications have been established, but no industry-scale application exists to build energy forecasts. Accordingly, this study is the world’s first to establish a transformer-based model to estimate the energy consumption of a real-scale university library and benchmark with a baseline model (Support Vector Regression) SVR. With a large dataset from 1 September 2017 to 13 November 2021 with 30 min granularity, the results using four historical electricity readings to estimate one future reading demonstrate that the SVR (an R2 of 0.92) presents superior performance than the transformer-based model (an R2 of 0.82). Across the sensitivity analysis, the SVR model is more sensitive to the input close to the output. These findings provide new insights into the research area of energy forecasting in either a specific building or a building cluster in a city. The influences of the number of inputs and outputs related to the transformer-based model will be investigated in the future.

Texture aware autoencoder pre-training and pairwise learning refinement for improved iris recognition

This paper presents a texture aware end-to-end trainable iris recognition system. We build upon our previous stagewise learning framework but present two key contributions: a) we propose a better autoencoding framework with a data relation loss between Gram matrix representations of input and reconstructed images. The data relation loss enables learning better texture representation which is pivotal for a texture rich dataset such as iris. Robustness of auto-encoding is further enhanced with an auxiliary denoising task. b) we design a pairwise learning architecture which subsumes the task of iris matching inside the training pipeline itself and results in significant improvement in matching performance compared to usual offline matching paradigm. On ND-IRIS-0405, CASIA.v4-Interval and IITD iris datasets our proposed model achieves better matching performance over both traditional baselines and recent deep learning paradigms. Specifically, our method yields a relative improvement of 42.30%, 64.92% and 20% in terms of equal error rates (EER) with respect to the best competing deep learning method on the respective datasets.

AI-oriented Smart Power System Transient Stability: The Rationality, Applications, Challenges and Future Opportunities

Nowadays, the power grid has become an active colossal resource generation and management system due to the wide use of renewable energy and dynamic workloads processed through intelligent information and communication technologies. Several new operations exist, such as power electrification, intelligent information integration on the physical layer, and complex interconnections in the smart grid. These procedures use data-driven deep learning, big data, and machine learning paradigms to efficiently analyze and control electric power system transient problems and resolve technical issues with robust accuracy and timeliness. Thus, artificial intelligence (AI) has become vital to address and resolving issues related to transient stability assessment (TSA) and control generation. In this paper, we provide a comprehensive review on the role of AI and its sub-procedures in addressing problems in TSA. The workflow of the article includes an AI-based intelligent power system structure along with power system TSA and AI-application rationality to transient situations. Outperforms other reviews, this paper discusses the AI-based TSA framework and design process along with intelligent applications and their analytics in power system transient problems. Moreover, we are not limited to AI, but we also combine the direction of big data that is highly compatible with AI, discusses future trends, opportunities, challenges, and open issues of AI-Big data based transient stability assessment in the smart power grid.

Reducing bi-level feature redundancy for unsupervised domain adaptation

Unsupervised domain adaptation (UDA) deals with the problem of transferring knowledge from a labeled source domain to an unlabeled target domain when the two domains have distinct data distributions. Therefore, the purpose of domain adaptation is to mitigate the distribution divergence between the two domains. Many existing UDA methods only use the traditional batch normalization layer, but this may lead to a large number of feature redundancy and lead performance degradation. In this paper, we introduce a novel deep learning paradigm called feature redundancy in UDA to enhance adaptation ability. Specifically, we first show that feature redundancy also exists on unsupervised domain adaptation (UDA), which has been ignored by most previous efforts. We utilize feature similarity as a metric to measure feature redundancy and then analyze the relationship between uniform feature spectrum and minimal feature similarity. Based on this relationship, we intend to reduce cross-domain feature redundancy for UDA by making the distribution of feature spectrum uniforms in a bi-level way. For the first level, we propose a cross-domain batch normalization with the whitening module (xBN) to ensure compact domain-specific features and learn domain-invariant features at the same time. With the domain-specific features from the first level that paves a way, on the second level, we suggest an alternative orthogonal regularizer (OR) that can make the distribution of the feature spectrum more uniform, thus domain-invariant feature redundancy is mitigated. Such a bi-level mechanism greatly reduces the feature redundancy for UDA. To evaluate the efficacy of the proposed bi-level mechanism, we plug those two novel modules (i.e., xBN and OR) into convolutional neural networks (CNNs) to form our UDA model and also conduct the corresponding empirical evaluations on five cross-domain object recognition benchmarks including both classical and large-scale image datasets. Experimental results show that the proposed UDA model could achieve state-of-the-art performance both in quantity and quality. Our source codes will be released after publication.

Node Intrusion Tendency Recognition Using Network Level Features Based Deep Learning Approach

Adhoc network is highly susceptible for intrusion attacks due to the simplified accesscontrol and compacted network stack. Malicious node recognition in Mobile adhocnetwork (MANET) is challengeable due to nodes mobility and limited coverageof nodes. Thus, link may keep fluctuating throughout the communication period.In this paper, deep analytic model is made for extracting attacker node behaviorsfrom networking point of view. Attributed such as link durations, re-healing timeand number of received packets (by attacker) was the main features of this work.Later, deep learning paradigm is integrated to perform attacker node recognition. Dataobtained from network analytical model is used to train three different models namelyFeed forward neural network (FFNN), Cascade backpropagation neural network(CBPNN) and Convolutional neural network (CNN). Attacker node recognitionaccuracy of 85.5