Multiscale Vision Research Articles

Hybrid Convolution-Based Efficientnet-Based Hand Gesture Recognition Framework with Optimized Algorithm

The difficulties in communication and hearing are an important concern for deaf–dumb people, which stop access to their essential and basic needs. Many findings have been made to address sign languages even though this challenging problem is not still solved. Many methods aimed to propose vision-based classifiers through identical pattern investigation tasks by obtaining the difficult handcraft feature descriptions of gestures from the gathered images. However, the efficacy of all those models is less for performing with a huge signbook captured from uncontrolled and complex background conditions. So, an effective Indian Sign Language (ISL) classification method is developed by an advanced deep learning approach. At first, the hand gesture images are obtained from the data source. Only the image of the hand, even from a complicated background, is extracted from the obtained image. The features are extracted using the Scale-Invariant Feature Transform (SIFT) method and Multiscale Vision Transformer (MVT). Then, the extracted features are fed to the Hybrid Convolution-based EfficientNet (HCEN) model. The hyper-parameters in the developed HCEN model are tuned using the implemented Adaptive Political Optimizer (APO) algorithm. The recognized hand signs are obtained from the suggested HCEN model. Various experiments are conducted to determine the performance of the suggested deep learning-based hand gesture recognition model.

Multi-scale Vision Transformer toward improved non-invasive anaemia detection using palm video

Multi-scale Vision Transformer toward improved non-invasive anaemia detection using palm video

DiagSWin: A multi-scale vision transformer with diagonal-shaped windows for object detection and segmentation

Recently, Vision Transformer and its variants have demonstrated remarkable performance on various computer vision tasks, thanks to its competence in capturing global visual dependencies through self-attention. However, global self-attention suffers from high computational cost due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection and semantic segmentation). Many recent works have attempted to reduce the cost by applying fine-grained local attention, but these approaches cripple the long-range modeling power of the original self-attention mechanism. Furthermore, these approaches usually have similar receptive fields within each layer, thus limiting the ability of each self-attention layer to capture multi-scale features, resulting in performance degradation when handling images with objects of different scales. To address these issues, we develop the Diagonal-shaped Window (DiagSWin) attention mechanism for modeling attentions in diagonal regions at hybrid scales per attention layer. The key idea of DiagSWin attention is to inject multi-scale receptive field sizes into tokens: before computing the self-attention matrix, each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity. This mechanism is able to effectively capture multi-scale context information while reducing computational complexity. With DiagSwin attention, we present a new variant of Vision Transformer models, called DiagSWin Transformers, and demonstrate their superiority in extensive experiments across various tasks. Specifically, the DiagSwin Transformer with a large size achieves 84.4% Top-1 accuracy and outperforms the SOTA CSWin Transformer on ImageNet with 40% fewer model size and computation cost. When employed as backbones, DiagSWin Transformers achieve significant improvements over the current SOTA modules. In addition, our DiagSWin-Base model yields 51.1 box mAP and 45.8 mask mAP on COCO for object detection and segmentation, and 52.3 mIoU on the ADE20K for semantic segmentation.

Multimodal profiling of biostabilized human skin modules reveals a coordinated ecosystem response to injected mRNA-1273 COVID-19 vaccine.

The field of drug development is witnessing a remarkable surge in the development of innovative strategies. There is a need to develop technological platforms capable of generating human data prior to progressing to clinical trials. Here we introduce a new flexible solution designed for the comprehensive monitoring of the natural human skin ecosystem's response to immunogenic drugs over time. Based on unique bioengineering to preserve surgical resections in a long survival state, it allows for the first time a comprehensive analysis of resident immune cells response at both organ and single-cell levels. Upon injection of the mRNA-1273 COVID-19 vaccine, we characterized precise sequential molecular events triggered upon detection of the exogenous substance. The vaccine consistently targets DC/macrophages and mast cells, regardless of the administration route, while promoting specific cell-cell communications in surrounding immune cell subsets. Given its direct translational relevance, this approach provides a multiscale vision of genuine human tissue immunity that could pave the way toward the development of new vaccination and drug development strategies.

A novel maternal thyroid disease prediction using multi-scale vision transformer architecture with improved linguistic hedges neural-fuzzy classifier.

Early pregnancy thyroid function assessment in mothers is covered. The benefits of using load-specific reference ranges are well-established. We pondered whether the categorization of maternal thyroid function would change if multiple blood samples obtained early in pregnancy were used. Even though binary classification is a common goal of current disease diagnosis techniques, the data sets are small, and the outcomes are not validated. Most current approaches concentrate on model optimization, focusing less on feature engineering. The suggested method can predict increased protein binding, non-thyroid syndrome (NTIS) (simultaneous non-thyroid disease), autoimmune thyroiditis (compensated hypothyroidism), and Hashimoto's thyroiditis (primary hypothyroidism). In this paper, we develop an automatic thyroid nodule classification system using a multi-scale vision transformer and image enhancement. Graph equalization is the chosen technique for image enhancement, and in our experiments, we used neural networks with four-layer network nodes. This work presents an enhanced linguistic coverage neuro-fuzzy classifier with chosen features for thyroid disease feature selection diagnosis. The training procedure is optimized, and a multi-scale vision transformer network is employed. Each hop connection in Dense Net now has trainable weight parameters, altering the architecture. Images of thyroid nodules from 508 patients make up the data set for this article. Sets of 80% training and 20% validation and 70% training and 30% validation are created from the data. Simultaneously, we take into account how the number of training iterations, network structure, activation function of network nodes, and other factors affect the classification outcomes. According to the experimental results, the best number of training iterations is 500, the logistic function is the best activation function, and the ideal network structure is 2500-40-2-1. K-fold validation and performance comparison with previous research validate the suggested methodology's enhanced effectiveness.

TransFusion: Efficient Vision Transformer based on 3D transesophageal echocardiography images for the left atrial appendage segmentation

3D transesophageal echocardiography (TEE) is widely used in the preoperative guidance of left atrial appendage closure (LAAC), and is the preferred imaging examination recommended by expert consensus. The precise extraction of the left atrial appendage (LAA) is the essential initial step for establishing an automated predictive procedure of the LAAC. However, due to the inherent limitations of the ultrasound images and the unique morphological features of the LAA, segmenting the LAA in TEE images is a challenging task. In this paper, we propose a novel Transformer-based dual feature fusion network (TransFusion) for LAA segmentation in TEE images. Our TransFusion includes both intra-stage and inter-stage multi-scale feature fusions. Specifically, in each stage of the encoder, we introduce a Multi-Scale Vision Transformer (MSViT) block for extracting and enhancing multi-scale feature representations. Additionally, we use a Feature Fusion module (FFM) to fuse the features across different stages in parallel and alleviate the semantic gap and better recovering the spatial details. Finally, the convolutional neural network (CNN) as the decoder is used to output the results of LAA segmentation. We compare our method with several state-of-the-art methods on our private clinical dataset. Experimental results demonstrate the superior robustness and segmentation performance of our method and its capability to accurately extract the LAA.

TransMVAN: Multi-view Aggregation Network with Transformer for Pneumonia Diagnosis.

Automated and accurate classification of pneumonia plays a crucial role in improving the performance of computer-aided diagnosis systems for chest X-ray images. Nevertheless, it is a challenging task due to the difficulty of learning the complex structure information of lung abnormality from chest X-ray images. In this paper, we propose a multi-view aggregation network with Transformer (TransMVAN) for pneumonia classification in chest X-ray images. Specifically, we propose to incorporate the knowledge from glance and focus views to enrich the feature representation of lung abnormality. Moreover, to capture the complex relationships among different lung regions, we propose a bi-directional multi-scale vision Transformer (biMSVT), with which the informative messages between different lung regions are propagated through two directions. In addition, we also propose a gated multi-view aggregation (GMVA) to adaptively select the feature information from glance and focus views for further performance enhancement of pneumonia diagnosis. Our proposed method achieves AUCs of 0.9645 and 0.9550 for pneumonia classification on two different chest X-ray image datasets. In addition, it achieves an AUC of 0.9761 for evaluating positive and negative polymerase chain reaction (PCR). Furthermore, our proposed method also attains an AUC of 0.9741 for classifying non-COVID-19 pneumonia, COVID-19 pneumonia, and normal cases. Experimental results demonstrate the effectiveness of our method over other methods used for comparison in pneumonia diagnosis from chest X-ray images.

Use of a Multiscale Vision Transformer to Predict Nursing Activities Score From Low-Resolution Thermal Videos in an Intensive Care Unit

Use of a Multiscale Vision Transformer to Predict Nursing Activities Score From Low-Resolution Thermal Videos in an Intensive Care Unit

Computer-aided colorectal cancer diagnosis: AI-driven image segmentation and classification

Colorectal cancer is an enormous health concern since it is among the most lethal types of malignancy. The manual examination has its limitations, including subjectivity and data overload. To overcome these challenges, computer-aided diagnostic systems focusing on image segmentation and abnormality classification have been developed. This study presents a two-stage approach for the automatic detection of five types of colorectal abnormalities in addition to a control group: polyp, low-grade intraepithelial neoplasia, high-grade intraepithelial neoplasia, serrated adenoma, adenocarcinoma. In the first stage, UNet3+ was used for image segmentation to locate the anomalies, while in the second stage, the Cross-Attention Multi-Scale Vision Transformer deep learning model was used to predict the type of anomaly after highlighting the anomaly on the raw images. In anomaly segmentation, UNet3+ achieved values of 0.9872, 0.9422, 0.9832, and 0.9560 for Dice Coefficient, Jaccard Index, Sensitivity, Specificity respectively. In anomaly detection, the Cross-Attention Multi-Scale Vision Transformer model attained a classification performance of 0.9340, 0.9037, 0.9446, 0.8723, 0.9102, 0.9849 for accuracy, F1 score, precision, recall, Matthews correlation coefficient, and specificity, respectively. The proposed approach proves its capacity to alleviate the overwhelm of pathologists and enhance the accuracy of colorectal cancer diagnosis by achieving high performance in both the identification of anomalies and the segmentation of regions.

Conditional generative adversarial networks for image-based sunlight analysis of residential blocks

Sunlight simulation is often used to analyze whether buildings receive sufficient sunlight in complex design cases. However, it is time-consuming to establish geometrical and physical models, and non-negligible computation time (from a few seconds to minutes) leads to high time cost of iterative design optimization. Therefore, this study builds surrogate models to instantaneously detect problematic buildings (those receiving insufficient sunlight) using images of residential blocks. Different conditional generative adversarial networks (CGANs), including Pix2Pix and conditional attention GAN (CAGAN), are employed to train surrogate models. The attention mechanisms and the multi-scale vision give CAGAN an edge in capturing the global structure of buildings and learning high-level knowledge related to the occlusion relationships between buildings. In addition, this study also proposes a stepwise strategy to construct composite surrogate models. The first sub-model provides more informative images to help the second sub-model learn the features of problematic buildings. This study focuses on evaluating the models based on the detection performance in addition to image structural similarity and pixel similarity, which have been used in most studies. The CAGAN models outperform the Pix2Pix models in the average detection accuracy (ADA) and the 100 % successful detection probability (SDP) on the same dataset. Moreover, the stepwise strategy increases the ADA and SDP to 89.88 % and 71.07 %, respectively. The proposed methods can be extended to other urban environmental performance diagnoses, such as thermal comfort or the wind environment.

End-to-End Ultrasonic Hand Gesture Recognition.

As the number of electronic gadgets in our daily lives is increasing and most of them require some kind of human interaction, this demands innovative, convenient input methods. There are limitations to state-of-the-art (SotA) ultrasound-based hand gesture recognition (HGR) systems in terms of robustness and accuracy. This research presents a novel machine learning (ML)-based end-to-end solution for hand gesture recognition with low-cost micro-electromechanical (MEMS) system ultrasonic transducers. In contrast to prior methods, our ML model processes the raw echo samples directly instead of using pre-processed data. Consequently, the processing flow presented in this work leaves it to the ML model to extract the important information from the echo data. The success of this approach is demonstrated as follows. Four MEMS ultrasonic transducers are placed in three different geometrical arrangements. For each arrangement, different types of ML models are optimized and benchmarked on datasets acquired with the presented custom hardware (HW): convolutional neural networks (CNNs), gated recurrent units (GRUs), long short-term memory (LSTM), vision transformer (ViT), and cross-attention multi-scale vision transformer (CrossViT). The three last-mentioned ML models reached more than 88% accuracy. The most important innovation described in this research paper is that we were able to demonstrate that little pre-processing is necessary to obtain high accuracy in ultrasonic HGR for several arrangements of cost-effective and low-power MEMS ultrasonic transducer arrays. Even the computationally intensive Fourier transform can be omitted. The presented approach is further compared to HGR systems using other sensor types such as vision, WiFi, radar, and state-of-the-art ultrasound-based HGR systems. Direct processing of the sensor signals by a compact model makes ultrasonic hand gesture recognition a true low-cost and power-efficient input method.

A vision transformer machine learning model for COVID-19 diagnosis using chest X-ray images

This study leverages machine learning to enhance the diagnostic accuracy of COVID-19 using chest X-rays. The study evaluates various architectures, including efficient neural networks (EfficientNet), multiscale vision transformers (MViT), efficient vision transformers (EfficientViT), and vision transformers (ViT), against a comprehensive open-source dataset comprising 3616 COVID-19, 6012 lung opacity, 10192 normal, and 1345 viral pneumonia images. The analysis, focusing on loss functions and evaluation metrics, demonstrates distinct performance variations among these models. Notably, multiscale models like MViT and EfficientNet tend towards overfitting. Conversely, our vision transformer model, innovatively fine-tuned (FT) on the encoder blocks, exhibits superior accuracy: 95.79% in four-class, 99.57% in three-class, and similarly high performance in binary classifications, along with a recall of 98.58%, precision of 98.87%, F1 score of 98.73%, specificity of 99.76%, and area under the receiver operating characteristic (ROC) curve (AUC) of 0.9993. The study confirms the vision transformer model’s efficacy through rigorous validation using quantitative metrics and visualization techniques and illustrates its superiority over conventional models. The innovative fine-tuning method applied to vision transformers presents a significant advancement in medical image analysis, offering a promising avenue for improving the accuracy and reliability of COVID-19 diagnosis from chest X-ray images.

Large-scale assessment of date palm plantations based on UAV remote sensing and multiscale vision transformer

Timely and efficient mapping of date palm plantations through unmanned aerial vehicle (UAV) remote sensing is critical for continuous observation, health and risk evaluation, pest management, resource optimization, and ensuring the long-term sustainability of the dates industry. This study presents an efficient and cost-effective transformer-based approach to identify, countify, monitor, and evaluate the overall well-being of palm trees using extensive UAV imagery. The suggested approach integrates an improved multiscale vision transformer, feature pyramid network, Mask R–CNN, and improved slicing-aided hyper inference for practical large-scale assessments. This combination enabled the extraction of multiscale features, capturing long-range dependencies in the data and boosting the model's generalizability. The proposed architecture outperformed several CNN-based architectures (including Mask R–CNN, Cascade Mask R–CNN, Point-based Rendering, and You Only Look At CoefficientTs), achieving F-scores of 94.33% and 94.2% for date palm tree detection and segmentation, respectively. The transformer-based architecture was optimized using transfer learning to differentiate between healthy and unhealthy date palm trees, particularly those with severe infestations. The potential generic condition of date palm trees was predicted with an F-score of 88.4%. Further advancements in this field could pave the way for a proactive strategy, enabling timely detection, which would aid in pest management and support the sustainable growth of the dates sector.

SingleS2R: Single sample driven Sim-to-Real transfer for Multi-Source Visual-Tactile Information Understanding using multi-scale vision transformers

Due to variations in light transmission and wear on the contact head, existing visual-tactile dataset building methods typically require a large amount of real-world data, making the dataset building process time-consuming and labor-intensive. Sim-to-Real learning has been proposed to realize Multi-Source Visual-Tactile Information Understanding (MSVTIU) in simulate and real environment, which can efficiently promote visual-tactile dataset building using simulation method for emerged robotic applications. However, the existing Sim-to-Real learning also requires more than 10,000 real data, while the corresponding data need to be re-collected when the sensor version changes. To address this challenge, we propose a powerful Sim-to-Real transfer for MSVTIU which requires only one single real-world tactile sample. To effectively extract features from the single real tactile sample, a multi-scale vision transformers-based Generative Adversarial Network (GAN) is proposed to address MSVTIU task under extremely limited data. We introduce a novel scale-dependent self-attention mechanism that allows attention layers to adapt their behavior at different stages of the generating process. In addition, we introduced a residual block for capturing contextual information between adjacent scales, which utilizes shortcut connections to fully preserve texture and structure information. We subsequently enhanced the model understanding of visual-tactile information using elastic transform and adaptive adversarial training strategy, both of which are designed specifically for MSVTIU. Experiments on two public datasets with diverse objects indicate that our Sim-to-Real transfer approach utilizing only one single real-world visual-tactile sample outperforms the state-of-the-art methods that requires tens of thousands of samples.

Solar Flare Forecasting Based on Magnetogram Sequences Learning with Multiscale Vision Transformers and Data Augmentation Techniques

Solar Flare Forecasting Based on Magnetogram Sequences Learning with Multiscale Vision Transformers and Data Augmentation Techniques

MgMViT: Multi-Granularity and Multi-Scale Vision Transformer for Efficient Action Recognition

Nowadays, the field of video-based action recognition is rapidly developing. Although Vision Transformers (ViT) have made great progress in static image processing, they are not yet fully optimized for dynamic video applications. Convolutional Neural Networks (CNN) and related models perform exceptionally well in video action recognition. However, there are still some issues that cannot be ignored, such as high computational costs and large memory consumption. In the face of these issues, current research focuses on finding effective methods to improve model performance and overcome current limits. Therefore, we present a unique Vision Transformer model based on multi-granularity and multi-scale fusion to accomplish efficient action recognition, which is designed for action recognition in videos to effectively reduce computational costs and memory usage. Firstly, we devise a multi-scale, multi-granularity module that integrates with Transformer blocks. Secondly, a hierarchical structure is utilized to manage information at various scales, and we introduce multi-granularity on top of multi-scale, which allows for a selective choice of the number of tokens to enter the next computational step, thereby reducing redundant tokens. Thirdly, a coarse-fine granularity fusion layer is introduced to reduce the sequence length of tokens with lower information content. The above two mechanisms are combined to optimize the allocation of resources in the model, further emphasizing critical information and reducing redundancy, thereby minimizing computational costs. To assess our proposed approach, comprehensive experiments are conducted by using benchmark datasets in the action recognition domain. The experimental results demonstrate that our method has achieved state-of-the-art performance in terms of accuracy and efficiency.

A Joint Convolutional Cross ViT Network for Hyperspectral and Light Detection and Ranging Fusion Classification

The fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data for classification has received widespread attention and has led to significant progress in research and remote sensing applications. However, existing common CNN architectures suffer from the significant drawback of not being able to model remote sensing images globally, while transformer architectures are not able to capture local features effectively. To address these bottlenecks, this paper proposes a classification framework for multisource remote sensing image fusion. First, a spatial and spectral feature projection network is constructed based on parallel feature extraction by combining HSI and LiDAR data, which is conducive to extracting joint spatial, spectral, and elevation features from different source data. Furthermore, in order to construct local–global nonlinear feature mapping more flexibly, a network architecture coupling together multiscale convolution and a multiscale vision transformer is proposed. Moreover, a plug-and-play nonlocal feature token aggregation module is designed to adaptively adjust the domain offsets between different features, while a class token is employed to reduce the complexity of high-dimensional feature fusion. On three open-source remote sensing datasets, the performance of the proposed multisource fusion classification framework improves about 1% to 3% over other state-of-the-art algorithms.

Complex-Valued Multiscale Vision Transformer on Space Target Recognition by ISAR Image Sequence

Complex-Valued Multiscale Vision Transformer on Space Target Recognition by ISAR Image Sequence

MSVTNet: Multi-Scale Vision Transformer Neural Network for EEG-Based Motor Imagery Decoding.

Transformer-based neural networks have been applied to the electroencephalography (EEG) decoding for motor imagery (MI). However, most networks focus on applying the self-attention mechanism to extract global temporal information, while the cross-frequency coupling features between different frequencies have been neglected. Additionally, effectively integrating different neural networks poses challenges for the advanced design of decoding algorithms. This study proposes a novel end-to-end Multi-Scale Vision Transformer Neural Network (MSVTNet) for MI-EEG classification. MSVTNet first extracts local spatio-temporal features at different filtered scales through convolutional neural networks (CNNs). Then, these features are concatenated along the feature dimension to form local multi-scale spatio-temporal feature tokens. Finally, Transformers are utilized to capture cross-scale interaction information and global temporal correlations, providing more distinguishable feature embeddings for classification. Moreover, auxiliary branch loss is leveraged for intermediate supervision to ensure the effective integration of CNNs and Transformers. The performance of MSVTNet was assessed through subject-dependent (session-dependent and session-independent) and subject-independent experiments on three MI datasets, i.e., the BCI competition IV 2a, 2b and OpenBMI datasets. The experimental results demonstrate that MSVTNet achieves state-of-the-art performance in all analyses. MSVTNet shows superiority and robustness in enhancing MI decoding performance. The source code for MSVTNet is available at https://github.com/SheepTAO/MSVTNet.

What is the in-host dynamics of the SARS-CoV-2 virus? A challenge within a multiscale vision of living systems

This paper deals with the modeling and simulation of the in-host dynamics of a virus. The modeling approach was developed according to the idea that mathematical models should go beyond deterministic single-scale population dynamics by taking into account the multiscale, heterogeneous features of the complex system under consideration. Here, we considered modeling the competition between the virus, the epithelial cells it infects, and the heterogeneous immune system with evolving activation states that induce a range of different effects on virus particles and infected cells. The subsequent numerical simulations showed different types of model outcomes: from virus elimination, to virus persistence and periodic relapse, to virus uncontrolled growth that triggers a blow-up in the fully activated immune response. The simulations also showed the existence of a threshold in the immune response that separates the regimes of higher re-infections from lower re-infections (compared to the magnitude of the first viral infection).