Convolution-based Model Research Articles

Extremely compact video representation for efficient near-duplicates detection

With the influx of online video uploads, a robust method for detecting video copies under various distortions is essential for copyright protection and search efficiency. We propose a compact video representation that uses a Siamese Neural Network for near-duplicate detection. This approach achieves up to 0.998 recall and 0.853 precision, even with distorted 56x56px miniatures. Consequently, we derive extremely compact video descriptors, facilitating video fragment retrieval in large datasets. Our method pre-selects frames from a video by identifying local maximums from the interframe differences curve, preserving characteristic sequence patterns even after extreme compression. Utilizing a simpler convolution-based model in the Siamese Neural Network, we improve results by up to 8% over VGG-16, with a 1.7-fold reduction in inference time. With compact descriptors of 1.875 kB, our models outperform others by more than two times. Additionally, we introduce a new dataset with dynamically changing scenes, enhancing its suitability for near-duplicate video detection.

A dual-scale fused hypergraph convolution-based hyperedge prediction model for predicting missing reactions in genome-scale metabolic networks.

Genome-scale metabolic models (GEMs) are powerful tools for predicting cellular metabolic and physiological states. However, there are still missing reactions in GEMs due to incomplete knowledge. Recent gaps filling methods suggest directly predicting missing responses without relying on phenotypic data. However, they do not differentiate between substrates and products when constructing the prediction models, which affects the predictive performance of the models. In this paper, we propose a hyperedge prediction model that distinguishes substrates and products based on dual-scale fused hypergraph convolution, DSHCNet, for inferring the missing reactions to effectively fill gaps in the GEM. First, we model each hyperedge as a heterogeneous complete graph and then decompose it into three subgraphs at both homogeneous and heterogeneous scales. Then we design two graph convolution-based models to, respectively, extract features of the vertices in two scales, which are then fused via the attention mechanism. Finally, the features of all vertices are further pooled to generate the representative feature of the hyperedge. The strategy of graph decomposition in DSHCNet enables the vertices to engage in message passing independently at both scales, thereby enhancing the capability of information propagation and making the obtained product and substrate features more distinguishable. The experimental results show that the average recovery rate of missing reactions obtained by DSHCNet is at least 11.7% higher than that of the state-of-the-art methods, and that the gap-filled GEMs based on our DSHCNet model achieve the best prediction performance, demonstrating the superiority of our method.

EndoSRR: a comprehensive multi-stage approach for endoscopic specular reflection removal.

Specular reflections in endoscopic images not only disturb visual perception but also hamper computer vision algorithm performance. However, the intricate nature and variability of these reflections, coupled with a lack of relevant datasets, pose ongoing challenges for removal. We present EndoSRR, a robust method for eliminating specular reflections in endoscopic images. EndoSRR comprises two stages: reflection detection and reflection region inpainting. In the reflection detection stage, we adapt and fine-tune the segment anything model (SAM) using a weakly labeled dataset, achieving an accurate reflection mask. For reflective region inpainting, we employ LaMa, a fast Fourier convolution-based model trained on a 4.5M-image dataset, enabling effective inpainting of arbitrarily shaped reflection regions. Lastly, we introduce an iterative optimization strategy for dual pre-trained models to refine the results of specular reflection removal, named DPMIO. Utilizing the SCARED-2019 dataset, our approach surpasses state-of-the-art methods in both qualitative and quantitative evaluations. Qualitatively, our method excels in accurately detecting reflective regions, yielding more natural and realistic inpainting results. Quantitatively, our method demonstrates superior performance in both segmentation evaluation metrics (IoU, E-measure, etc.) and image inpainting evaluation metrics (PSNR, SSIM, etc.). The experimental results underscore the significance of proficient endoscopic specular reflection removal for enhancing visual perception and downstream tasks. The methodology and results presented in this study are poised to catalyze advancements in specular reflection removal, thereby augmenting the accuracy and safety of minimally invasive surgery.

Improving the Adversarial Transferability of Vision Transformers with Virtual Dense Connection

With the great achievement of vision transformers (ViTs), transformer-based approaches have become the new paradigm for solving various computer vision tasks. However, recent research shows that similar to convolutional neural networks (CNNs), ViTs are still vulnerable to adversarial attacks. To explore the shared deficiency of models with different structures, researchers begin to analyze the cross-structure adversarial transferability, which is still under-explored. Therefore, in this work, we focus on the ViT attacks to improve the cross-structure transferability between the transformer-based and convolution-based models. Previous studies fail to thoroughly investigate the influence of the components inside the ViT models on adversarial transferability, leading to inferior performance. To overcome the drawback, we launch a motivating study by linearly down-scaling the gradients of components inside the ViT models to analyze their influence on adversarial transferability. Based on the motivating study, we find that the gradient of the skip connection most influences transferability and believe that back-propagating gradients from deeper blocks can enhance transferability. Therefore, we propose the Virtual Dense Connection method (VDC). Specifically, without changing the forward pass, we first recompose the original network to add virtual dense connections. Then we back-propagate gradients of deeper Attention maps and Multi-layer Perceptron (MLP) blocks via virtual dense connections when generating adversarial samples. Extensive experiments confirm the superiority of our proposed method over the state-of-the-art baselines, with an 8.2% improvement in transferability between ViT models and a 7.2% improvement in cross-structure transferability from ViTs to CNNs.

E(n) Equivariant Graph Neural Network for Learning Interactional Properties of Molecules.

We have developed a multi-input E(n) equivariant graph convolution-based model designed for the prediction of chemical properties that result from the interaction of heterogeneous molecular structures. By incorporating spatial features and constraining the functions learned from these features to be equivariant to E(n) symmetries, the interactional-equivariant graph neural network (IEGNN) can efficiently learn from the 3D structure of multiple molecules. To verify the IEGNN's capability to learn interactional properties, we tested the model's performance on three molecular data sets, two of which are curated in this study and made publicly available for future interactional model benchmarking. To enable the loading of these data sets, an open-source data structure based on the PyTorch Geometric library for batch loading multigraph data points is also created. Finally, the IEGNN's performance on a data set consisting of an unknown interactional relationship (the frictional properties resulting between monolayers with variable composition) is examined. The IEGNN model developed was found to have the lowest mean absolute percent error for the predicted tribological properties of four of the six data sets when compared to previous methods.

BeatProfiler: Multimodal In Vitro Analysis of Cardiac Function Enables Machine Learning Classification of Diseases and Drugs.

Goal: Contractile response and calcium handling are central to understanding cardiac function and physiology, yet existing methods of analysis to quantify these metrics are often time-consuming, prone to mistakes, or require specialized equipment/license. We developed BeatProfiler, a suite of cardiac analysis tools designed to quantify contractile function, calcium handling, and force generation for multiple in vitro cardiac models and apply downstream machine learning methods for deep phenotyping and classification. Methods: We first validate BeatProfiler's accuracy, robustness, and speed by benchmarking against existing tools with a fixed dataset. We further confirm its ability to robustly characterize disease and dose-dependent drug response. We then demonstrate that the data acquired by our automatic acquisition pipeline can be further harnessed for machine learning (ML) analysis to phenotype a disease model of restrictive cardiomyopathy and profile cardioactive drug functional response. To accurately classify between these biological signals, we apply feature-based ML and deep learning models (temporal convolutional-bidirectional long short-term memory model or TCN-BiLSTM). Results: Benchmarking against existing tools revealed that BeatProfiler detected and analyzed contraction and calcium signals better than existing tools through improved sensitivity in low signal data, reduction in false positives, and analysis speed increase by 7 to 50-fold. Of signals accurately detected by published methods (PMs), BeatProfiler's extracted features showed high correlations to PMs, confirming that it is reliable and consistent with PMs. The features extracted by BeatProfiler classified restrictive cardiomyopathy cardiomyocytes from isogenic healthy controls with 98% accuracy and identified relax90 as a top distinguishing feature in congruence with previous findings. We also show that our TCN-BiLSTM model was able to classify drug-free control and 4 cardiac drugs with different mechanisms of action at 96% accuracy. We further apply Grad-CAM on our convolution-based models to identify signature regions of perturbations by these drugs in calcium signals. Conclusions: We anticipate that the capabilities of BeatProfiler will help advance in vitro studies in cardiac biology through rapid phenotyping, revealing mechanisms underlying cardiac health and disease, and enabling objective classification of cardiac disease and responses to drugs.

Are transformer-based models more robust than CNN-based models?

As the deployment of artificial intelligence (AI) models in real-world settings grows, their open-environment robustness becomes increasingly critical. This study aims to dissect the robustness of deep learning models, particularly comparing transformer-based models against CNN-based models. We focus on unraveling the sources of robustness from two key perspectives: structural and process robustness. Our findings suggest that transformer-based models generally outperform convolution-based models in robustness across multiple metrics. However, we contend that these metrics may not wholly represent true model robustness, such as the mean of corruption error. To better understand the underpinnings of this robustness advantage, we analyze models through the lens of Fourier transform and game interaction. From our insights, we propose a calibrated evaluation metric for robustness against real-world data, and a blur-based method to enhance robustness performance. Our approach achieves state-of-the-art results, with mCE scores of 2.1% on CIFAR-10-C, 12.4% on CIFAR-100-C, and 24.9% on TinyImageNet-C.

Dynamic and Static Handwriting Assessment in Parkinson's Disease: A Synergistic Approach with C-Bi-GRU and VGG19

Parkinson's disease (PD) is a neurodegenerative disorder causing a decline in dopamine levels, impacting the peripheral nervous system and motor functions. Current detection methods often identify PD at advanced stages. This study addresses early-stage detection using handwriting analysis, specifically exploring the PaHaW dataset for pen pressure and stroke movement data. Evaluating online and offline features, the research employs pre-trained CNN models (VGG 19 and AlexNet) for offline datasets, achieving an overall accuracy of 0.53. For online datasets, velocity, and acceleration features are extracted and classified using Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), and recurrent neural networks (RNN), with GRU yielding the highest accuracy at 0.57. Notably, the convolution-based model C-Bi-GRU surpasses other architectures with a remarkable 0.75 accuracy, emphasizing its efficacy in early PD detection. These findings underscore the potential of handwriting analysis as a diagnostic tool for PD, contributing valuable insights for further research and development in medical diagnostics.

IntME: Combined Improving Feature Interactions and Matrix Multiplication for Convolution-Based Knowledge Graph Embedding

Knowledge graph embedding is a method to predict missing links in knowledge graphs by learning the interactions between embedded entities and relations in a continuous low-dimensional space. Current research on convolution-based models tends to provide sufficient interactions for extracting potential knowledge. However, sufficient interactions do not mean that they are reasonable. Our studies find that reasonable interactions can further stimulate knowledge extraction capability. Reasonable interactions need to ensure that the elements participating in interactions are disordered and in a reasonable number. To model reasonable interactions that cannot be specifically quantified, we propose a concise and effective model IntME to address this challenge. In detail, we utilize checked feature reshaping and disordered matrix multiplication to form two different types of feature maps to ensure the disorder of the interacting elements and control the number of elements before feature fusion by the shapes of the feature maps after channel scaling reshaping. In feature fusion, we employ large convolution filters and pointwise filters for the deep and shallow linear fusion of feature interactions, which can take into account both explicit and implicit knowledge extraction capability. The evaluations of four benchmark datasets show that IntME has a powerful performance in convolution-based models and a lower training cost, and also demonstrate that our proposed approaches based on reasonable interactions can effectively improve knowledge discovery capability.

Deep learning for deterioration prediction of COVID-19 patients based on time-series of three vital signs

Unrecognized deterioration of COVID-19 patients can lead to high morbidity and mortality. Most existing deterioration prediction models require a large number of clinical information, typically collected in hospital settings, such as medical images or comprehensive laboratory tests. This is infeasible for telehealth solutions and highlights a gap in deterioration prediction models based on minimal data, which can be recorded at a large scale in any clinic, nursing home, or even at the patient’s home. In this study, we develop and compare two prognostic models that predict if a patient will experience deterioration in the forthcoming 3 to 24 h. The models sequentially process routine triadic vital signs: (a) oxygen saturation, (b) heart rate, and (c) temperature. These models are also provided with basic patient information, including sex, age, vaccination status, vaccination date, and status of obesity, hypertension, or diabetes. The difference between the two models is the way that the temporal dynamics of the vital signs are processed. Model #1 utilizes a temporally-dilated version of the Long-Short Term Memory model (LSTM) for temporal processes, and Model #2 utilizes a residual temporal convolutional network (TCN) for this purpose. We train and evaluate the models using data collected from 37,006 COVID-19 patients at NYU Langone Health in New York, USA. The convolution-based model outperforms the LSTM based model, achieving a high AUROC of 0.8844–0.9336 for 3 to 24 h deterioration prediction on a held-out test set. We also conduct occlusion experiments to evaluate the importance of each input feature, which reveals the significance of continuously monitoring the variation of the vital signs. Our results show the prospect for accurate deterioration forecast using a minimum feature set that can be relatively easily obtained using wearable devices and self-reported patient information.

Modelling constituent order despite symmetric associations in memory

Mathematical models of association memory (study AB, given A, recall B) either predict that knowledge for constituent order of a word pair (AB vs. BA) is perfectly unrelated, or completely dependent on knowledge of the pairing itself. Data contradict both predictions; when a pair is remembered, constituent-order is above chance, but still fairly low. Convolution-based models are inherently symmetric and can explain associative symmetry, but cannot discriminate AB from BA. We evaluated four extensions of convolution, where order is incorporated as item features, partial permutations of features, item-position associations, or by adding item and position vectors. All approaches could discriminate order within behaviourally observed ranges, without compromising associative symmetry. Only the permutation model could disambiguate AB from BC in double-function lists, as humans can do. It is possible that each of our proposed mechanisms might apply to a different, particular task setting. However, the partial permutation model can thus far explain the broadest set of empirical benchmarks.

A deep-learning framework for spray pattern segmentation and estimation in agricultural spraying systems

This work focuses on leveraging deep learning for agricultural applications, especially for spray pattern segmentation and spray cone angle estimation. These two characteristics are important to understanding the sprayer system such as nozzles used in agriculture. The core of this work includes three deep-learning convolution-based models. These models are trained and their performances are compared. After the best model is selected based on its performance, it is used for spray region segmentation and spray cone angle estimation. The output from the selected model provides a binary image representing the spray region. This binary image is further processed using image processing to estimate the spray cone angle. The validation process is designed to compare results obtained from this work with manual measurements.

Short human eccDNAs are predictable from sequences.

Ubiquitous presence of short extrachromosomal circular DNAs (eccDNAs) in eukaryotic cells has perplexed generations of biologists. Their widespread origins in the genome lacking apparent specificity led some studies to conclude their formation as random or near-random. Despite this, the search for specific formation of short eccDNA continues with a recent surge of interest in biomarker development. To shed new light on the conflicting views on short eccDNAs' randomness, here we present DeepCircle, a bioinformatics framework incorporating convolution- and attention-based neural networks to assess their predictability. Short human eccDNAs from different datasets indeed have low similarity in genomic locations, but DeepCircle successfully learned shared DNA sequence features to make accurate cross-datasets predictions (accuracy: convolution-based models: 79.65 ± 4.7%, attention-based models: 83.31 ± 4.18%). The excellent performance of our models shows that the intrinsic predictability of eccDNAs is encoded in the sequences across tissue origins. Our work demonstrates how the perceived lack of specificity in genomics data can be re-assessed by deep learning models to uncover unexpected similarity.

Dynamic compact thermal models for skin temperature prediction of portable electronic devices based on convolution and fitting methods

In most portable electronic devices, besides the temperature of multiple heat sources, i.e. junction temperature, the temperature of the enclosure, i.e. skin temperature, should also be controlled to protect the user experience. Thus, generating the device-level compact thermal model for predicting the skin temperature will not only improve the efficiency of early-stage thermal design but also help to develop the model-based temperature control strategy. However, most of the existing modeling methods mainly focus on predicting the junction temperature. In this paper, the dynamic compact thermal models of two portable electronic devices, including a smartphone and laptop, are first generated based on the convolution method. Under the assumption of linear time-invariant (LTI) systems, the skin temperature of the two test devices could be fast calculated once the step response of each heat source is obtained. The results show that compared to the computational fluid dynamics (CFD) model generated in ANSYS Icepak, the transient deviation of the convolution-based model is within 5%. Then, a linear fitting model is proposed to real-time predict the skin temperature of laptops. Both the experimental and simulation results demonstrate that once the scale factors are trained, the linear fitting model could accurately real-time predict the skin temperature after 60s. The maximum transient deviation in the experiment and simulation are 4% and 7% respectively. The results indicate that the proposed modeling method has tremendous potential for both thermal design and control optimization.

Convolution-based approach for modeling the paliperidone extended release and Long-Acting Injectable (LAI) PK of once-, and three-monthly products administration and for optimizing the development of new LAI products

The aim of this paper was to develop a convolution-based modeling approach for describing the paliperidone PK resulting from the administration of extended-release once-a-day oral dose, and once- and three monthly long-acting injectable products and to compare the performances of this approach to the traditional modeling strategy. The results of the analyses indicated that the traditional and convolution-based models showed comparable performances in the characterization of the paliperidone PK. However, the convolution-based approach showed several appealing features that justify the choice of this modeling as a preferred tool for modeling Long Acting Injectable (LAI) products and for deploying an effective model-informed drug development process. In particular, the convolution-based modeling can (a) facilitate the development of in vitro/in vivo correlation, (b) be used to identify formulations with optimal in vivo release properties, and (c) be used for optimizing the clinical benefit of a treatment by supporting the implementation of integrated models connecting in vitro and in vivo drug release, in vivo drug release to PK, and PK to PD and biomarker endpoints. A case study was presented to illustrate the benefits and the flexibility of the convolution-based modeling outcomes. The model was used to evaluate the in vivo drug release properties associated with a hypothetical once-a-year administration of a LAI product with the assumption that the expected paliperidone exposure during a 3-year treatment overlays the exposure expected after repeated administrations of a 3-month LAI product.

A spatiotemporal graph convolution-based model for daily runoff prediction in a river network with non-Euclidean topological structure

A spatiotemporal graph convolution-based model for daily runoff prediction in a river network with non-Euclidean topological structure

Rethinking Design and Evaluation of 3D Point Cloud Segmentation Models

Currently, the use of 3D point clouds is rapidly increasing in many engineering fields, such as geoscience and manufacturing. Various studies have developed intelligent segmentation models providing accurate results, while only a few of them provide additional insights into the efficiency and robustness of their proposed models. The process of segmentation in the image domain has been studied to a great extent and the research findings are tremendous. However, the segmentation analysis with point clouds is considered particularly challenging due to their unordered and irregular nature. Additionally, solving downstream tasks with 3D point clouds is computationally inefficient, as point clouds normally consist of thousands or millions of points sparsely distributed in 3D space. Thus, there is a significant need for rigorous evaluation of the design characteristics of segmentation models, to be effective and practical. Consequently, in this paper, an in-depth analysis of five fundamental and representative deep learning models for 3D point cloud segmentation is presented. Specifically, we investigate multiple experimental dimensions, such as accuracy, efficiency, and robustness in part segmentation (ShapeNet) and scene segmentation (S3DIS), to assess the effective utilization of the models. Moreover, we create a correspondence between their design properties and experimental properties. For example, we show that convolution-based models that incorporate adaptive weight or position pooling local aggregation operations achieve superior accuracy and robustness to point-wise MLPs, while the latter ones show higher efficiency in time and memory allocation. Our findings pave the way for an effective 3D point cloud segmentation model selection and enlighten the research on point clouds and deep learning.

Robust R-peak detection in an electrocardiogram with stationary wavelet transformation and separable convolution

R-peak detection is an essential step in analyzing electrocardiograms (ECGs). Previous deep learning models reported their performance primarily in a single database, and some models did not perform at the highest levels when applied to a database different from the testing database. To achieve high performances in cross-database validations, we developed a novel deep learning model for R-peak detection using stationary wavelet transform (SWT) and separable convolution. Three databases (i.e., the MIT-BIH Arrhythmia [MIT-BIH], the Institute of Cardiological Technics [INCART], and the QT) were used in both the training and testing models, and the MIT-BIH ST Change (MIT-BIH-ST), European ST-T, TELE and MIT-BIH Noise Stress Test (MIT-BIH-NST) databases were further used for testing. The detail coefficient of level 4 decomposition by SWT and the first derivative from filtered ECGs were used for model inputs, and the interval of 150 ms centered at marked peaks was used for labels. Separable convolution with atrous spatial pyramidal pooling was selected as the model’s architecture, and noise-augmented waveforms of 5.69 s duration (2048 size in 360 Hz) were used in training. The model performance was evaluated using cross-database validation. The F1 scores of the peak detection model were 0.9994, 0.9985, and 0.9999 in the MIT-BIH, INCART, and QT databases, respectively. When the above three databases were pooled, the F1 scores were 0.9993 for fivefold cross-validation and 0.9991 for cross-database validation. The model performance remained high for MIT-BIH-ST, European ST-T, and TELE, with F1 scores of 0.9995, 0.9988, and 0.9790, respectively. The model performance when trained by severe noise augmentation increased for the MIT-BIH-NST database (F1 scores from 0.9504 to 0.9759) and decreased for the MIT-BIH database (F1 scores from 0.9994 to 0.9991). The present SWT and separable convolution-based model for R-peak detection yields a high performance even for cross-database validations.

Detection of COVID-19 Infection in CT and X-ray images using transfer learning approach.

The infection caused by the SARS-CoV-2 (COVID-19) pandemic is a threat to human lives. An early and accurate diagnosis is necessary for treatment. The study presents an efficient classification methodology for precise identification of infection caused by COVID-19 using CT and X-ray images. The depthwise separable convolution-based model of MobileNet V2 was exploited for feature extraction. The features of infection were supplied to the SVM classifier for training which produced accurate classification results. The accuracies for CT and X-ray images are 99.42% and 98.54% respectively. The MCC score was used to avoid any mislead caused by accuracy and F1 score as it is more mathematically balanced metric. The MCC scores obtained for CT and X-ray were 0.9852 and 0.9657, respectively. The Youden's index showed a significant improvement of more than 2% for both imaging techniques. The proposed transfer learning-based approach obtained the best results for all evaluation metrics and produced reliable results for the accurate identification of COVID-19 symptoms. This study can help in reducing the time in diagnosis of the infection.

Transformer-Based Model with Dynamic Attention Pyramid Head for Semantic Segmentation of VHR Remote Sensing Imagery.

Convolutional neural networks have long dominated semantic segmentation of very-high-resolution (VHR) remote sensing (RS) images. However, restricted by the fixed receptive field of convolution operation, convolution-based models cannot directly obtain contextual information. Meanwhile, Swin Transformer possesses great potential in modeling long-range dependencies. Nevertheless, Swin Transformer breaks images into patches that are single-dimension sequences without considering the position loss problem inside patches. Therefore, Inspired by Swin Transformer and Unet, we propose SUD-Net (Swin transformer-based Unet-like with Dynamic attention pyramid head Network), a new U-shaped architecture composed of Swin Transformer blocks and convolution layers simultaneously through a dual encoder and an upsampling decoder with a Dynamic Attention Pyramid Head (DAPH) attached to the backbone. First, we propose a dual encoder structure combining Swin Transformer blocks and reslayers in reverse order to complement global semantics with detailed representations. Second, aiming at the spatial loss problem inside each patch, we design a Multi-Path Fusion Model (MPFM) with specially devised Patch Attention (PA) to encode position information of patches and adaptively fuse features of different scales through attention mechanisms. Third, a Dynamic Attention Pyramid Head is constructed with deformable convolution to dynamically aggregate effective and important semantic information. SUD-Net achieves exceptional results on ISPRS Potsdam and Vaihingen datasets with 92.51%mF1, 86.4%mIoU, 92.98%OA, 89.49%mF1, 81.26%mIoU, and 90.95%OA, respectively.