Branch Convolutional Neural Network Research Articles

DDNet: Depth Dominant Network for Semantic Segmentation of RGB-D Images.

Convolutional neural networks (CNNs) have been widely applied to parse indoor scenes and segment objects represented by color images. Nonetheless, the lack of geometric and context information is a problem for most RGB-based methods, with which depth features are only used as an auxiliary module in RGB-D semantic segmentation. In this study, a novel depth dominant network (DDNet) is proposed to fully utilize the rich context information in the depth map. The critical insight is that obvious geometric information from the depth image is more conducive to segmentation than RGB data. Compared with other methods, DDNet is a depth-based network with two branches of CNNs to extract color and depth features. As the core of the encoder network, the depth branch is given a larger fusion weight to extract geometric information, while semantic information and complementary geometric information are provided by the color branch for the depth feature maps. The effectiveness of our proposed depth-based architecture has been demonstrated by comprehensive experimental evaluations and ablation studies on challenging RGB-D semantic segmentation benchmarks, including NYUv2 and a subset of ScanNetv2.

Deep learning-based authentication for insider threat detection in critical infrastructure

In today’s cyber environment, threats such as data breaches, cyberattacks, and unauthorized access threaten national security, critical infrastructure, and financial stability. This research addresses the challenging task of protecting critical infrastructure from insider threats because of the high level of trust and access these individuals typically receive. Insiders may obtain a system administrator’s password through close observation or by deploying software to gather the information. To solve this issue, an innovative artificial intelligence-based methodology is proposed to identify a user by their password’s keystroke dynamics. This paper also introduces a new Gabor Filter Matrix Transformation method to transform numerical values into images by revealing the behavioral pattern of password typing. A siamese neural network (SNN) with the branches of convolutional neural networks is utilized for image comparison, aiming to detect unauthorized attempts to access critical infrastructure systems. The network analyzes the unique features of a user’s password timestamps transformed into images and compares them with previously submitted user passwords. The obtained results indicate that transforming the numerical values of keystroke dynamics into images and training an SNN leads to a lower equal error rate (EER) and higher user authentication accuracy than those previously reported in other studies. The methodology is validated on publicly available keystroke dynamics collections, the CMU and GREYC-NISLAB datasets, which collectively comprise over 30,000 password samples. It achieves the lowest EER value of 0.04545 compared to state-of-the-art methods for transforming non-image data into images. The paper concludes with a discussion of findings and potential future directions.

Spatial–Temporal Analysis-Based Video Quality Assessment: A Two-Stream Convolutional Network Approach

In system processing, video inevitably suffers from distortion, which leads to quality degradation and affects the user experience. Therefore, it is of great importance to design an accurate and effective objective video quality assessment (VQA) method. In this paper, by considering the multi-dimensional characteristics for video and visual perceptual mechanism, a two-stream convolutional network for VQA is proposed based on spatial–temporal analysis, named TSCNN-VQA. Specifically, for feature extraction, TSCNN-VQA first extracts spatial and temporal features by two different convolutional neural network branches, respectively. After that, the spatial–temporal joint feature fusion is constructed to obtain the joint spatial–temporal features. Meanwhile, the TSCNN-VQA also integrates an attention module to guarantee that the process conforms to the mechanism that the visual system perceives video information. Finally, the overall quality is obtained by non-linear regression. The experimental results in both the LIVE and CSIQ VQA datasets show that the performance indicators obtained by TSCNN-VQA are higher than those of existing VQA methods, which demonstrates that TSCNN-VQA can accurately evaluate video quality and has better consistency with the human visual system.

CSAP-UNet: Convolution and self-attention paralleling network for medical image segmentation with edge enhancement

Convolution operation is performed within a local window of the input image. Therefore, convolutional neural network (CNN) is skilled in obtaining local information. Meanwhile, the self-attention (SA) mechanism extracts features by calculating the correlation between tokens from all positions in the image, which has advantage in obtaining global information. Therefore, the two modules can complement each other to improve feature extraction ability. An effective fusion method is a problem worthy of further study. In this paper, we propose a CNN and SA paralleling network CSAP-UNet with U-Net as backbone. The encoder consists of two parallel branches of CNN and Transformer to extract the feature from the input image, which takes into account both the global dependencies and the local information. Because medical images come from certain frequency bands within the spectrum, their color channels are not as uniform as natural images. Meanwhile, medical segmentation pays more attention to lesion regions in the image. Attention fusion module (AFM) integrates channel attention and spatial attention in series to fuse the output features of the two branches. The medical image segmentation task is essentially to locate the boundary of the object in the image. The boundary enhancement module (BEM) is designed in the shallow layer of the proposed network to focus more specifically on pixel-level edge details. Experimental results on three public datasets validate that CSAP-UNet outperforms state-of-the-art networks, particularly on the ISIC 2017 dataset. The cross-dataset evaluation on Kvasir and CVC-ClinicDB shows that CSAP-UNet has strong generalization ability. Ablation experiments also indicate the effectiveness of the designed modules. The code for training and test is available at https://github.com/zhouzhou1201/CSAP-UNet.git.

Collaborative Encoding Method for Scene Text Recognition in Low Linguistic Resources: The Uyghur Language Case Study

Current research on scene text recognition primarily focuses on languages with abundant linguistic resources, such as English and Chinese. In contrast, there is relatively limited research dedicated to low-resource languages. Advanced methods for scene text recognition often employ Transformer-based architectures. However, the performance of Transformer architectures is suboptimal when dealing with low-resource datasets. This paper proposes a Collaborative Encoding Method for Scene Text Recognition in the low-resource Uyghur language. The encoding framework comprises three main modules: the Filter module, the Dual-Branch Feature Extraction module, and the Dynamic Fusion module. The Filter module, consisting of a series of upsampling and downsampling operations, performs coarse-grained filtering on input images to reduce the impact of scene noise on the model, thereby obtaining more accurate feature information. The Dual-Branch Feature Extraction module adopts a parallel structure combining Transformer encoding and Convolutional Neural Network (CNN) encoding to capture local and global information. The Dynamic Fusion module employs an attention mechanism to dynamically merge the feature information obtained from the Transformer and CNN branches. To address the scarcity of real data for natural scene Uyghur text recognition, this paper conducted two rounds of data augmentation on a dataset of 7267 real images, resulting in 254,345 and 3,052,140 scene images, respectively. This process partially mitigated the issue of insufficient Uyghur language data, making low-resource scene text recognition research feasible. Experimental results demonstrate that the proposed collaborative encoding approach achieves outstanding performance. Compared to baseline methods, our collaborative encoding approach improves accuracy by 14.1%.

CVTrack: Combined Convolutional Neural Network and Vision Transformer Fusion Model for Visual Tracking.

Most single-object trackers currently employ either a convolutional neural network (CNN) or a vision transformer as the backbone for object tracking. In CNNs, convolutional operations excel at extracting local features but struggle to capture global representations. On the other hand, vision transformers utilize cascaded self-attention modules to capture long-range feature dependencies but may overlook local feature details. To address these limitations, we propose a target-tracking algorithm called CVTrack, which leverages a parallel dual-branch backbone network combining CNN and Transformer for feature extraction and fusion. Firstly, CVTrack utilizes a parallel dual-branch feature extraction network with CNN and transformer branches to extract local and global features from the input image. Through bidirectional information interaction channels, the local features from the CNN branch and the global features from the transformer branch are able to interact and fuse information effectively. Secondly, deep cross-correlation operations and transformer-based methods are employed to fuse the template and search region features, enabling comprehensive interaction between them. Subsequently, the fused features are fed into the prediction module to accomplish the object-tracking task. Our tracker achieves state-of-the-art performance on five benchmark datasets while maintaining real-time execution speed. Finally, we conduct ablation studies to demonstrate the efficacy of each module in the parallel dual-branch feature extraction backbone network.

SUTrans-NET: a hybrid transformer approach to skin lesion segmentation.

Melanoma is a malignant skin tumor that threatens human life and health. Early detection is essential for effective treatment. However, the low contrast between melanoma lesions and normal skin and the irregularity in size and shape make skin lesions difficult to detect with the naked eye in the early stages, making the task of skin lesion segmentation challenging. Traditional encoder-decoder built with U-shaped networks using convolutional neural network (CNN) networks have limitations in establishing long-term dependencies and global contextual connections, while the Transformer architecture is limited in its application to small medical datasets. To address these issues, we propose a new skin lesion segmentation network, SUTrans-NET, which combines CNN and Transformer in a parallel fashion to form a dual encoder, where both CNN and Transformer branches perform dynamic interactive fusion of image information in each layer. At the same time, we introduce our designed multi-grouping module SpatialGroupAttention (SGA) to complement the spatial and texture information of the Transformer branch, and utilize the Focus idea of YOLOV5 to construct the Patch Embedding module in the Transformer to prevent the loss of pixel accuracy. In addition, we design a decoder with full-scale information fusion capability to fully fuse shallow and deep features at different stages of the encoder. The effectiveness of our method is demonstrated on the ISIC 2016, ISIC 2017, ISIC 2018 and PH2 datasets and its advantages over existing methods are verified.

A novel two-stream multi-head self-attention convolutional neural network for bearing fault diagnosis

Bearing is one of the most prone to failure components, and the most commonly used supporting parts in rotating machinery. There are important economic and safety significance for studying effective bearing fault diagnosis method. Deep learning has been widely applied to intelligent fault diagnosis of rotating machines. In particular, convolutional neural network (CNN) improves diagnostic accuracy due to automatic feature extraction ability. However, CNN cannot fully use the temporal information of the input signals and extract global features. This article presents a new depth model called two-stream multi-head self-attention convolutional neural network (TSMSCNN) for bearing fault diagnosis to solve the above problems. First, we obtain temporal information through the difference of data points in the input signal. Next, the temporal information and original signal are used as input for two CNN branches. Then, the multi-head self-attention (MS) mechanism adaptively aggregates the features and assigns weights to the features extracted by CNN. Finally, the learned high-level representations are fed into the full connect (FC) layer for fault diagnosis. Experimental results prove that the proposed TSMSCNN has higher accuracy than state-of-the-art methods and has excellent noise immunity and transfer performance.

Unobtrusive hand gesture recognition using ultra-wide band radar and deep learning

<span lang="EN-US">Hand function after stroke injuries is not regained rapidly and requires physical rehabilitation for at least 6 months. Due to the heavy burden on the healthcare system, assisted rehabilitation is prescribed for a limited time, whereas so-called home rehabilitation is offered. It is therefore essential to develop robust solutions that facilitate monitoring while preserving the privacy of patients in a home-based setting. To meet these expectations, an unobtrusive solution based on radar sensing and deep learning is proposed. The multi-input multi-output convolutional eXtra trees (MIMO-CxT) is a new deep hybrid model used for hand gesture recognition (HGR) with impulse-radio ultra-wide band (IR-UWB) radars. It consists of a lightweight architecture based on a multi-input convolutional neural network (CNN) used in a hybrid configuration with extremely randomized trees (ETs). The model takes data from multiple sensors as input and processes them separately. The outputs of the CNN branches are concatenated before the prediction is made by the ETs. Moreover, the model uses depthwise separable convolution layers, which reduce computational cost and learning time while maintaining high performance. The model is evaluated on a publicly available dataset of gestures collected by three IR-UWB radars and achieved an average accuracy of 98.86%.</span>

BrainGridNet: A two-branch depthwise CNN for decoding EEG-based multi-class motor imagery

Brain–computer interfaces (BCIs) based on motor imagery (MI) enable the disabled to interact with the world through brain signals. To meet demands of real-time, stable, and diverse interactions, it is crucial to develop lightweight networks that can accurately and reliably decode multi-class MI tasks. In this paper, we introduce BrainGridNet, a convolutional neural network (CNN) framework that integrates two intersecting depthwise CNN branches with 3D electroencephalography (EEG) data to decode a five-class MI task. The BrainGridNet attains competitive results in both the time and frequency domains, with superior performance in the frequency domain. As a result, an accuracy of 80.26 percent and a kappa value of 0.753 are achieved by BrainGridNet, surpassing the state-of-the-art (SOTA) model. Additionally, BrainGridNet shows optimal computational efficiency, excels in decoding the most challenging subject, and maintains robust accuracy despite the random loss of 16 electrode signals. Finally, the visualizations demonstrate that BrainGridNet learns discriminative features and identifies critical brain regions and frequency bands corresponding to each MI class. The convergence of BrainGridNet’s strong feature extraction capability, high decoding accuracy, steady decoding efficacy, and low computational costs renders it an appealing choice for facilitating the development of BCIs.

Regional feature learning using attribute structural analysis in bipartite attention framework for vehicle re-identification

<p><span lang="EN-US">Vehicle re-identification identifies target vehicles using images obtained by numerous non-overlapping real-time surveillance cameras. The effectiveness of re-identification is further challenging because of illumination changes, pose differences of captured images, and resolution. Fine-grained appearance changes in vehicles are recognized in addition to the coarse-grained characteristics like color of the vehicle along with model, and other custom features like logo stickers, annual service signs, and hangings to overcome these challenges. To prove the efficiency of our proposed bipartite attention framework, a novel dataset called Attributes27 which has 27 labelled attributes for each class are created. Our framework contains three major sections: The first section where the overall and semantic characteristics of every individual vehicle image are extracted by a double branch convolutional neural network (CNN) layer. Secondly, to identify the region of interests (ROIs) each branch has a self-attention block linked to it. Lastly to extract the regional features from the obtained ROIs, a partition-alignment block is deployed. The results of our proposed system’s evaluation on the Attributes27 and VeRi-776 datasets has highlighted significant regional attributes of each vehicle and improved the accuracy. Attributes27 and VeRi-776 datasets exhibits 98.5% and 84.3% accuracy respectively which are comparatively higher than the existing methods with 78.6% accuracy.</span></p>

DeepAFP: An effective computational framework for identifying antifungal peptides based on deep learning.

Fungal infections have become a significant global health issue, affecting millions worldwide. Antifungal peptides (AFPs) have emerged as a promising alternative to conventional antifungal drugs due to their low toxicity and low propensity for inducing resistance. In this study, we developed a deep learning-based framework called DeepAFP to efficiently identify AFPs. DeepAFP fully leverages and mines composition information, evolutionary information, and physicochemical properties of peptides by employing combined kernels from multiple branches of convolutional neural network with bi-directional long short-term memory layers. In addition, DeepAFP integrates a transfer learning strategy to obtain efficient representations of peptides for improving model performance. DeepAFP demonstrates strong predictive ability on carefully curated datasets, yielding an accuracy of 93.29% and an F1-score of 93.45% on the DeepAFP-Main dataset. The experimental results show that DeepAFP outperforms existing AFP prediction tools, achieving state-of-the-art performance. Finally, we provide a downloadable AFP prediction tool to meet the demands of large-scale prediction and facilitate the usage of our framework by the public or other researchers. Our framework can accurately identify AFPs in a short time without requiring significant human and material resources, and hence can accelerate the development of AFPs as well as contribute to the treatment of fungal infections. Furthermore, our method can provide new perspectives for other biological sequence analysis tasks.

Fruit ripeness identification using YOLOv8 model

Deep learning-based visual object detection is a fundamental aspect of computer vision. These models not only locate and classify multiple objects within an image, but they also identify bounding boxes. The focus of this paper's research work is to classify fruits as ripe or overripe using digital images. Our proposed model extracts visual features from fruit images and analyzes fruit peel characteristics to predict the fruit's class. We utilize our own datasets to train two "anchor-free" models: YOLOv8 and CenterNet, aiming to produce accurate predictions. The CenterNet network primarily incorporates ResNet-50 and employs the deconvolution module DeConv for feature map upsampling. The final three branches of convolutional neural networks are applied to predict the heatmap. The YOLOv8 model leverages CSP and C2f modules for lightweight processing. After analyzing and comparing the two models, we found that the C2f module of the YOLOv8 model significantly enhances classification results, achieving an impressive accuracy rate of 99.5%.

TCANet: three-stream coordinate attention network for RGB-D indoor semantic segmentation

Semantic segmentation plays a vital role in indoor scene analysis. Currently, its accuracy is still limited due to the complex conditions of various indoor scenes. In addition, it is difficult to complete this task solely relying on RGB images. Since depth images can provide additional 3D geometric information to RGB images, researchers chose to incorporate depth images for improving the accuracy of indoor semantic segmentation. However, it is still a challenge to effectively fuse the depth information with the RGB images. To address this issue, a three-stream coordinate attention network is proposed. The presented network reconstructs a multi-modal feature fusion module for RGB-D features, which can realize the aggregation of two modal information along the spatial and channel dimensions. Meanwhile, three convolutional neural network branches are used to construct a parallel three-stream structure, which can, respectively, process the RGB features, depth features and combined features. On one hand, the proposed network can preserve the original RGB and depth feature streams, simultaneously. On the other hand, it can also contribute to utilize and propagate the fusion feature flow better. The embedded ASPP module is used to optimize the semantic information in the proposed network, so as to aggregate the feature information of different scales and obtain more accurate features. Experimental results show that the proposed model can reach a state-of-the-art mIoU accuracy of 50.2% on the NYUDv2 dataset and on the more complex SUN-RGBD dataset.

Combining transformers with CNN for multi-focus image fusion

Recently, deep convolutional neural network (CNN) based methods for multi-focus image fusion have achieved adequate performance. However, most of them cannot obtain spatially continuous results, especially in smooth regions and edges between focused and defocused regions. In this paper, we propose a novel end-to-end method, which merits both Transformers and CNNs, as a strong alternative for multi-focus image fusion task. Transformer has advantages over a CNN in that it can extract global features. It is able to make the fusion results to be spatially consistent. The proposed architecture consists of CNN and transformer branches, where transformer branches take feature map patches as inputs and leverages the transformer to propagate global contexts among patches. Moreover, in order to improve feature representation, we introduce online knowledge distillation learning strategy (KDL). The strategy achieves better interactions between global features and local features. Specifically, we design hard target and soft target by simply yet effectively ensembling outputs of two branches, which are used to supervise CNN and transformer branches. The experiments demonstrate the superiority of our proposed architecture and achieve competitive results with state-of-the-art methods.

An interpretable attention branch convolutional neural network for identifying geochemical anomalies related to mineralization

Geochemical anomalies play a crucial role in geochemical prospecting as they directly indicate the existence of mineralization. Deep learning algorithms (DLAs) have received increasing attention in the domain of multivariate geochemical anomaly recognition due to their extraordinary ability to extract complex and nonlinear geochemical patterns. However, the black-box nature inherent in DLAs typically prevents one from understanding the potential operational mechanisms behind the recognition procedure, resulting in lack of confidence in the identification results. In this study, visualization techniques and attention mechanisms were integrated into a convolutional neural network (CNN) as an attention branch in order to construct an interpretable attention branch convolutional neural network (ABCNN) model for geochemical anomaly identification. Specifically, the class activation mapping was applied to visualize the spatial features extracted by the model and to identify important regions during model inference. The attention mechanism was further incorporated into a CNN to generate attention maps for enhancing spatial features associated with mineralization and interpreting the recognition results based on attention weights. A case study for identifying multivariate geochemical anomalies linked to Au polymetallic mineralization in the western Henan Province of China was presented to illustrate and validate the feasibility of the ABCNN model. The geochemical anomalies delineated by the ABCNN model coincide well with known gold deposits and ore-controlling geological features. The comparative studies between the ABCNN and a traditional CNN further manifested that the incorporation of an attention mechanism can both effectively improve the model interpretability and performance.

DOA Estimation of Underwater Acoustic Array Signal Based on Wavelet Transform With Double Branch Convolutional Neural Network

In underwater acoustic communications, a hydrophone array is often used to receive the underwater acoustic signals to improve the gain of the received signal. The direction of arrival (DOA) estimation is a basic task in underwater acoustic array signal processing. In underwater signal transmission, due to the channel complexity, a large signal transmission loss, and heavy noise interference, the received signal is seriously distorted, and the DOA estimation is poor. In order to improve the accuracy of DOA estimation for underwater acoustic array signals, in this work, we propose continuous wavelet transform with convolutional neural network (CWT-CNN) method. We use a linear factor to improve the calculation process of continuous wavelet transform. The characteristics of time-frequency fusion of signals are calculated by constructing time-frequency array model with modified wavelet factors. The fused features are then used to train the proposed double branch CNN. As compared with other DOA algorithms based on neural networks, the proposed algorithm shows a 5%–20% higher accuracy and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$0.479^{\circ }-1.172^{\circ }$</tex-math></inline-formula> lower RMSE at 7 different values of SNR. In addition, the proposed algorithm is less computationally complex.

Multi‐branch sustainable convolutional neural network for disease classification

AbstractPandemic and natural disasters are growing more often, imposing even more pressure on life care services and users. There are knowledge gaps regarding how to prevent disasters and pandemics. In recent years, after heart disease, corona virus disease‐19 (COVID‐19), brain stroke, and cancer are at their peak. Different machine learning and deep learning‐based techniques are presented to detect these diseases. Existing technique uses two branches that have been used for detection and prediction of disease accurately such as brain hemorrhage. However, existing techniques have been focused on the detection of specific diseases with double‐branches convolutional neural networks (CNNs). There is a need to develop a model to detect multiple diseases at the same time using computerized tomography (CT) scan images. We proposed a model that consists of 12 branches of CNN to detect the different types of diseases with their subtypes using CT scan images and classify them more accurately. We proposed multi‐branch sustainable CNN model with deep learning architecture trained on the brain CT hemorrhage, COVID‐19 lung CT scans and chest CT scans with subtypes of lung cancers. Feature extracted automatically from preprocessed input data and passed to classifiers for classification in the form of concatenated feature vectors. Six classifiers support vector machine (SVM), decision tree (DT), K‐nearest neighbor (K‐NN), artificial neural network (ANN), naïve Bayes (NB), linear regression (LR) classifiers, and three ensembles the random forest (RF), AdaBoost, gradient boosting ensembles were tested on our model for classification and prediction. Our model achieved the best results on RF on each dataset. Respectively, on brain CT hemorrhage achieved (99.79%) accuracy, on COVID‐19 lung CT scans achieved (97.61%), and on chest CT scans dataset achieved (98.77%).

CTFuseNet: A Multi-Scale CNN-Transformer Feature Fused Network for Crop Type Segmentation on UAV Remote Sensing Imagery

Timely and accurate acquisition of crop type information is significant for irrigation scheduling, yield estimation, harvesting arrangement, etc. The unmanned aerial vehicle (UAV) has emerged as an effective way to obtain high resolution remote sensing images for crop type mapping. Convolutional neural network (CNN)-based methods have been widely used to predict crop types according to UAV remote sensing imagery, which has excellent local feature extraction capabilities. However, its receptive field limits the capture of global contextual information. To solve this issue, this study introduced the self-attention-based transformer that obtained long-term feature dependencies of remote sensing imagery as supplementary to local details for accurate crop-type segmentation in UAV remote sensing imagery and proposed an end-to-end CNN–transformer feature-fused network (CTFuseNet). The proposed CTFuseNet first provided a parallel structure of CNN and transformer branches in the encoder to extract both local and global semantic features from the imagery. A new feature-fusion module was designed to flexibly aggregate the multi-scale global and local features from the two branches. Finally, the FPNHead of feature pyramid network served as the decoder for the improved adaptation to the multi-scale fused features and output the crop-type segmentation results. Our comprehensive experiments indicated that the proposed CTFuseNet achieved a higher crop-type-segmentation accuracy, with a mean intersection over union of 85.33% and a pixel accuracy of 92.46% on the benchmark remote sensing dataset and outperformed the state-of-the-art networks, including U-Net, PSPNet, DeepLabV3+, DANet, OCRNet, SETR, and SegFormer. Therefore, the proposed CTFuseNet was beneficial for crop-type segmentation, revealing the advantage of fusing the features found by the CNN and the transformer. Further work is needed to promote accuracy and efficiency of this approach, as well as to assess the model transferability.

Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things

Human Activity Recognition (HAR) has become a crucial element for smart healthcare applications due to the fast adoption of wearable sensors and mobile technologies. Most of the existing human activity recognition frameworks deal with a single modality of data that degrades the reliability and recognition accuracy of the system for heterogeneous data sources. In this article, we propose a multi-level feature fusion technique for multimodal human activity recognition using multi-head Convolutional Neural Network (CNN) with Convolution Block Attention Module (CBAM) to process the visual data and Convolutional Long Short Term Memory (ConvLSTM) for dealing with the time-sensitive multi-source sensor information. The architecture is developed to be able to analyze and retrieve channel and spatial dimension features through the use of three branches of CNN along with CBAM for visual information. The ConvLSTM network is designed to capture temporal features from the multiple sensors’ time-series data for efficient activity recognition. An open-access multimodal HAR dataset named UP-Fall detection dataset is utilized in experiments and evaluations to measure the performance of the developed fusion architecture. Finally, we deployed an Internet of Things (IoT) system to test the proposed fusion network in real-world smart healthcare application scenarios. The findings from the experimental results reveal that the developed multimodal HAR framework surpasses the existing state-of-the-art methods in terms of multiple performance metrics.