Small Number Of Trainable Parameters Research Articles

Lung pneumonia severity scoring in chest X-ray images using transformers

To create robust and adaptable methods for lung pneumonia diagnosis and the assessment of its severity using chest X-rays (CXR), access to well-curated, extensive datasets is crucial. Many current severity quantification approaches require resource-intensive training for optimal results. Healthcare practitioners require efficient computational tools to swiftly identify COVID-19 cases and predict the severity of the condition. In this research, we introduce a novel image augmentation scheme as well as a neural network model founded on Vision Transformers (ViT) with a small number of trainable parameters for quantifying COVID-19 severity and other lung diseases. Our method, named Vision Transformer Regressor Infection Prediction (ViTReg-IP), leverages a ViT architecture and a regression head. To assess the model’s adaptability, we evaluate its performance on diverse chest radiograph datasets from various open sources. We conduct a comparative analysis against several competing deep learning methods. Our results achieved a minimum Mean Absolute Error (MAE) of 0.569 and 0.512 and a maximum Pearson Correlation Coefficient (PC) of 0.923 and 0.855 for the geographic extent score and the lung opacity score, respectively, when the CXRs from the RALO dataset were used in training. The experimental results reveal that our model delivers exceptional performance in severity quantification while maintaining robust generalizability, all with relatively modest computational requirements. The source codes used in our work are publicly available at https://github.com/bouthainas/ViTReg-IP.Graphical abstract

A robust neural network for privacy-preserving heart rate estimation in remote healthcare systems

In this study, we propose a computationally-light and robust neural network for estimating heart rate in remote healthcare systems. We develop a model that can be trained on consumer-grade graphics processing units (GPUs), and can be deployed on edge devices for swift inference. We propose a hybrid model based on convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) architectures for estimating heart rate from Electrocardiogram (ECG) and Photoplethysmography (PPG) signals. Considering the sensitive nature of the ECG signals, we ensure a formal privacy guarantee, differential privacy, for the model training. We perform a tight accounting of the overall privacy budget of our training algorithm using the Rényi Differential Privacy technique. We demonstrate that our model outperforms state-of-the-art networks on a benchmark dataset for both ECG and PPG signals despite having a much smaller number of trainable parameters and, consequently, much smaller training and inference times. Our CNN-BiLSTM architecture can also provide excellent heart rate estimation performance even under strict privacy constraints. We develop a prototype Arduino-based data collection system that is low-cost, efficient, and useful for providing access to modern healthcare services to people living in remote areas.

Advancing the Boundary of Pre-trained Models for Drug Discovery: Interpretable Fine-Tuning Empowered by Molecular Physicochemical Properties.

In the field of drug discovery, a proliferation of pre-trained models has surfaced, exhibiting exceptional performance across a variety of tasks. However, the extensive size of these models, coupled with the limited interpretative capabilities of current fine-tuning methods, impedes the integration of pre-trained models into the drug discovery process. This paper pushes the boundaries of pre-trained models in drug discovery by designing a novel fine-tuning paradigm known as the Head Feature Parallel Adapter (HFPA), which is highly interpretable, high-performing, and has fewer parameters than other widely used methods. Specifically, this approach enables the model to consider diverse information across representation subspaces concurrently by strategically using Adapters, which can operate directly within the model's feature space. Our tactic freezes the backbone model and forces various small-size Adapters' corresponding subspaces to focus on exploring different atomic and chemical bond knowledge, thus maintaining a small number of trainable parameters and enhancing the interpretability of the model. Moreover, we furnish a comprehensive interpretability analysis, imparting valuable insights into the chemical area. HFPA outperforms over seven physiology and toxicity tasks and achieves state-of-the-art results in three physical chemistry tasks. We also test ten additional molecular datasets, demonstrating the robustness and broad applicability of HFPA.

Agricultural weed identification in images and videos by integrating optimized deep learning architecture on an edge computing technology

Recent advancements in deep learning (DL)-based model optimization techniques have resulted in better weed identification accuracy. However, optimizing these models to identify weeds in images captured using small unmanned aerial system (UAS) has not been much explored. Moreover, leveraging the optimized model on resource constrained edge platform that could be easily integrated with UAS for real-time weed identification could be of significant advantage in developing precision aerial spraying weed management technology. Therefore, this study proposes YOLO-Spot model that is based on YOLOv7-tiny architecture, has been optimized and reconstructed to identify weeds amongst crop plants in aerial images and videos. The optimized model tends to use a smaller number of trainable parameters and reduced feature map sizes for weed identification. Most of the redundant convolutional layers along with feature map sizes have been reduced coupled with an integration of a novel module re-parameterized convolutional layer (RCL) within the neck component of the network. Furthermore, YOLO-Spot model has been trained on the three image resolutions, 320 × 320, 640 × 640 and 1280 × 1280, and has been named as YOLO-Spot_S, YOLO-Spot_M and YOLO-Spot_L, respectively. Out of all the variants, YOLO-Spot_M model has achieved significant prediction accuracy as compared to other variants and a denser layered model YOLOv7-Base. YOLO-Spot_M model utilizes over 75 % less parameters and 86 % reduced GFLOPs compared to YOLOv7-Base. As per the results, YOLO-Spot_M has outperformed YOLOv7-Base by achieving +1.3 % and +2.7 % overall accuracy and mAP(@0.5), respectively. The optimized architecture utilizes 4X less power (in W) when trained on a normal graphical processing unit (GPU). Moreover, converting YOLO-Spot_M to half-precision (FP16) for resource constrained device deployment (AGX Xavier), led to a +0.6 % accuracy and 5X faster weed recognition accuracy in aerial images and videos during inferencing. Based on the metrics obtained, YOLO-Spot_M model is recommended model that could be integrated with remote sensing technologies for site-specific weed management.

Multiple Instance Learning with Trainable Soft Decision Tree Ensembles

A new random forest-based model for solving the Multiple Instance Learning problem under small tabular data, called the Soft Tree Ensemble Multiple Instance Learning, is proposed. A new type of soft decision trees is considered, which is similar to the well-known soft oblique trees, but with a smaller number of trainable parameters. In order to train the trees, it is proposed to convert them into neural networks of a specific form, which approximate the tree functions. It is also proposed to aggregate the instance and bag embeddings (output vectors) by using the attention mechanism. The whole Soft Tree Ensemble Multiple Instance Learning model, including soft decision trees, neural networks, the attention mechanism and a classifier, is trained in an end-to-end manner. Numerical experiments with well-known real tabular datasets show that the proposed model can outperform many existing multiple instance learning models. A code implementing the model is publicly available.

Rapid training of quantum recurrent neural networks

Time series prediction is essential for human activities in diverse areas. A common approach to this task is to harness recurrent neural networks (RNNs). However, while their predictions are quite accurate, their learning process is complex and, thus, time and energy consuming. Here, we propose to extend the concept of RRNs by including continuous-variable quantum resources in it and to use a quantum-enhanced RNN to overcome these obstacles. The design of the continuous-variable quantum RNN (CV-QRNN) is rooted in the continuous-variable quantum computing paradigm. By performing extensive numerical simulations, we demonstrate that the quantum network is capable of learning-time dependence of several types of temporal data and that it converges to the optimal weights in fewer epochs than a classical network. Furthermore, for a small number of trainable parameters, it can achieve lower losses than its classical counterpart. CV-QRNN can be implemented using commercially available quantum-photonic hardware.

Canny-Net: Known Operator Learning for Edge Detection

We present Canny-Net, a modification of the 2D Canny edge detector into a pre-weighted neural network. CT scans of objects that contain metal components are characterized by artifacts like beam hardening, total absorption and scatter. Edge detection with classical methods, e.g. the Canny edge detector, is therefore prone to error. Using known operator learning, Canny-Net reduces maximum error bounds for edge detection in such CT scans. We show that Canny-Net is a light-weight neural network with a small number of trainable parameters. After training, we observe an increase of 11% in F1 score on 2304 test images when compared to the Canny edge detector. Due to its adaptability and low computational cost, Canny-Net can be considered for a wide range of different applications.

Exploring deep feature-blending capabilities to assist glaucoma screening

Over the last three decades, computer vision has had a vital role in the healthcare sector by providing soft computing-based robust and intelligent diagnostic solutions. Glaucoma is a critical ophthalmic disease that can trigger irreversible loss of vision. The number of patients with glaucoma is increasing dramatically worldwide. Manual ophthalmic assessment of glaucoma detection is a tedious, error-prone, time-consuming, and subjective task. Therefore, computer-assisted automatic glaucoma diagnosis methods are required to strengthen existing diagnostic methods with their robust performance. Optic disc (OD) and optic cup (OC) segmentation have a key role in glaucoma detection. Accurate segmentation of the OD and OC provides valuable computational and clinical details that can substantially assist in the glaucoma screening process. Retinal fundus images have extensive variations in terms of size, shape, pixel intensity values, and background effects that make segmentation challenging. To mitigate these challenges, we developed two novel networks for accurate OD and OC segmentation. An efficient shallow segmentation network (ESS-Net) is the base network whereas a feature-blending-based shallow segmentation network (FBSS-Net) is the final network of this work. ESS-Net is a shallow architecture with a maximum-depth semantic preservation block for accurate segmentation, while FBSS-Net uses internal and external feature blending to improve overall segmentation performance.To confirm their effectiveness, we evaluated both networks using four publicly available datasets; REFUGE, Drions-DB, Drishti-GS, and Rim-One-r3. The proposed methods exhibited excellent segmentation performance, requiring a small number of trainable parameters (3.02 million parameters).

Fast 3D Liver Segmentation Using a Trained Deep Chan-Vese Model

This paper introduces an approach for 3D organ segmentation that generalizes in multiple ways the Chan-Vese level set method. Chan-Vese is a segmentation method that simultaneously evolves a level set while fitting locally constant intensity models for the interior and exterior regions. First, its simple length-based regularization is replaced with a learned shape model based on a Fully Convolutional Network (FCN). We show how to train the FCN and introduce data augmentation methods to avoid overfitting. Second, two 3D variants of the method are introduced, one based on a 3D U-Net that makes global shape modifications and one based on a 3D FCN that makes local refinements. These two variants are integrated in a full 3D organ segmentation approach that is capable and efficient in dealing with the large size of the 3D volumes with minimal overfitting. Experiments on liver segmentation on a standard benchmark dataset show that the method obtains 3D segmentation results competitive with the state of the art while being very fast and having a small number of trainable parameters.

Tensor-Based Learning for Detecting Abnormalities on Digital Mammograms.

In this study, we propose a tensor-based learning model to efficiently detect abnormalities on digital mammograms. Due to the fact that the availability of medical data is limited and often restricted by GDPR (general data protection regulation) compliance, the need for more sophisticated and less data-hungry approaches is urgent. Accordingly, our proposed artificial intelligence framework utilizes the canonical polyadic decomposition to decrease the trainable parameters of the wrapped Rank-R FNN model, leading to efficient learning using small amounts of data. Our model was evaluated on the open source digital mammographic database INBreast and compared with state-of-the-art models in this domain. The experimental results show that the proposed solution performs well in comparison with the other deep learning models, such as AlexNet and SqueezeNet, achieving 90% ± 4% accuracy and an F1 score of 84% ± 5%. Additionally, our framework tends to attain more robust performance with small numbers of data and is computationally lighter for inference purposes, due to the small number of trainable parameters.

A lightweight deep learning model for retinal optical coherence tomography image classification

AbstractAutomated retinal disease detection and grading is one of the most researched areas in medical image analysis. In recent years, Deep Learning models have attracted much attention in this field. Hence, in this paper, we present a Deep Learning‐based, lightweight, fully automated end‐to‐end diagnostic system for the detection of the two major retinal diseases, namely diabetic macular oedema (DME) and drusen macular degeneration (DMD). Early detection of these diseases is important to prevent vision impairment. Optical coherence tomography (OCT) is the main imaging technique for detecting these diseases. The model proposed in this work is based on residual blocks and channel attention modules. The performance of the model is evaluated using the publicly available Mendeley OCT dataset and the Duke dataset. We were able to achieve a classification accuracy of 99.5% in the Mendeley test dataset and 94.9% in the Duke dataset with the proposed model. For the application, we performed an extensive evaluation of pre‐trained models (LeNet, AlexNet, VGG‐16, ResNet50 and SE‐ResNet). The proposed model has a much smaller number of trainable parameters and shows superior performance compared to existing methods.

Analysis of AI-Based Single-View 3D Reconstruction Methods for an Industrial Application.

Machine learning (ML) is a key technology in smart manufacturing as it provides insights into complex processes without requiring deep domain expertise. This work deals with deep learning algorithms to determine a 3D reconstruction from a single 2D grayscale image. The potential of 3D reconstruction can be used for quality control because the height values contain relevant information that is not visible in 2D data. Instead of 3D scans, estimated depth maps based on a 2D input image can be used with the advantage of a simple setup and a short recording time. Determining a 3D reconstruction from a single input image is a difficult task for which many algorithms and methods have been proposed in the past decades. In this work, three deep learning methods, namely stacked autoencoder (SAE), generative adversarial networks (GANs) and U-Nets are investigated, evaluated and compared for 3D reconstruction from a 2D grayscale image of laser-welded components. In this work, different variants of GANs are tested, with the conclusion that Wasserstein GANs (WGANs) are the most robust approach among them. To the best of our knowledge, the present paper considers for the first time the U-Net, which achieves outstanding results in semantic segmentation, in the context of 3D reconstruction tasks. Unlike the U-Net, which uses standard convolutions, the stacked dilated U-Net (SDU-Net) applies stacked dilated convolutions. Of all the 3D reconstruction approaches considered in this work, the SDU-Net shows the best performance, not only in terms of evaluation metrics but also in terms of computation time. Due to the comparably small number of trainable parameters and the suitability of the architecture for strong data augmentation, a robust model can be generated with only a few training data.

Modelling filtered reaction rate in turbulent premixed flames using feature importance analysis, gene expression programming and tiny artificial neural networks

Two different machine learning (ML) approaches, Symbolic regression, more precisely Gene Expression Programming (GEP) and Tiny Artificial Neural Networks (TANN), have been used for deducing a model for the filtered reaction rate in the context of Large Eddy Simulation (LES) of premixed combustion. The models have been trained on data generated from a large range of filter widths and for different turbulence intensities, using a simple chemistry Direct Numerical Simulation (DNS) database of statistically planar turbulent premixed flames. Feature importance analysis has been applied to identify the important input variables for modelling filtered reaction rate in order to enable an efficient optimization process as well as moderate inference times. The resulting models show excellent correlation and good quantitative agreement with the filtered reaction rates extracted from explicitly filtered DNS data. The genetic optimization results in a mathematical expression that can be easily implemented, analysed and documented. Also, the TANN has a small number of trainable parameters that can be shared with the research community. The present results demonstrate that ML algorithms can be successfully used for modelling turbulent premixed flames in the context of Large Eddy Simulation. Both approaches can fulfil the requirements of transparency and have inference times that are acceptable in comparison to algebraic models.

A new DDoS attacks intrusion detection model based on deep learning for cybersecurity

The data is exposed to many attacks during communication in the network environment. It is becoming increasingly essential to identify intrusions into network communications. Researchers use machine learning techniques to design effective intrusion detection systems. In this study, we proposed an intrusion detection system that includes preprocessing procedures and a deep learning model to detect DDoS attacks. For this purpose, various models based on Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), and Long Short Term Memory (LSTM) have been evaluated in terms of detection performance and real-time performance. We tested the suggested model using the CIC-DDoS2019 dataset, which is frequently used in the literature. We applied preprocess techniques such as feature elimination, random subset selection, feature selection, duplication removal, and normalization to the CIC-DDoS2019 dataset. As a result, better recognition performance was obtained for the training and testing evaluations. According to the test results, 99.99% for binary and 99.30% for multiclass accuracy using the CNN-based inception like model gave the best results among the proposed models. Also, the inference time of the proposed model for various sizes of test data looks promising compared to baseline models with a smaller number of trainable parameters. The proposed IDS system, together with the preprocessing methods, provides better results when compared to state-of-the-art studies.

Development of a Robust CNN Model for Capturing Microstructure-Property Linkages and Building Property Closures Supporting Material Design

Recent works have demonstrated the viability of convolutional neural networks (CNN) for capturing the highly non-linear microstructure-property linkages in high contrast composite material systems. In this work, we develop a new CNN architecture that utilizes a drastically reduced number of trainable parameters for building these linkages, compared to the benchmarks in current literature. This is accomplished by creating CNN architectures that completely avoid the use of fully connected layers, while using the 2-point spatial correlations of the microstructure as the input to the CNN. In addition to increased robustness (because of the much smaller number of trainable parameters), the CNN models developed in this work facilitate the construction of property closures at very low computational cost. This is because it allows for easy exploration of the space of valid 2-point spatial correlations, which is known to be a convex hull. Consequently, one can generate new sets of valid 2-point spatial correlations from previously available valid sets of 2-point spatial correlations, simply as convex combinations. This work demonstrates the significant benefits of utilizing 2-point spatial correlations as the input to the CNN, in place of the voxelated discrete microstructures used in current benchmarks.

Fruit classification using attention-based MobileNetV2 for industrial applications.

Recent deep learning methods for fruits classification resulted in promising performance. However, these methods are with heavy-weight architectures in nature, and hence require a higher storage and expensive training operations due to feeding a large number of training parameters. There is a necessity to explore lightweight deep learning models without compromising the classification accuracy. In this paper, we propose a lightweight deep learning model using the pre-trained MobileNetV2 model and attention module. First, the convolution features are extracted to capture the high-level object-based information. Second, an attention module is used to capture the interesting semantic information. The convolution and attention modules are then combined together to fuse both the high-level object-based information and the interesting semantic information, which is followed by the fully connected layers and the softmax layer. Evaluation of our proposed method, which leverages transfer learning approach, on three public fruit-related benchmark datasets shows that our proposed method outperforms the four latest deep learning methods with a smaller number of trainable parameters and a superior classification accuracy. Our model has a great potential to be adopted by industries closely related to the fruit growing and retailing or processing chain for automatic fruit identification and classifications in the future.

Detecting Blastocyst Components by Artificial Intelligence for Human Embryological Analysis to Improve Success Rate of In Vitro Fertilization.

Morphological attributes of human blastocyst components and their characteristics are highly correlated with the success rate of in vitro fertilization (IVF). Blastocyst component analysis aims to choose the most viable embryos to improve the success rate of IVF. The embryologist evaluates blastocyst viability by manual microscopic assessment of its components, such as zona pellucida (ZP), trophectoderm (TE), blastocoel (BL), and inner cell mass (ICM). With the success of deep learning in the medical diagnosis domain, semantic segmentation has the potential to detect crucial components of human blastocysts for computerized analysis. In this study, a sprint semantic segmentation network (SSS-Net) is proposed to accurately detect blastocyst components for embryological analysis. The proposed method is based on a fully convolutional semantic segmentation scheme that provides the pixel-wise classification of important blastocyst components that help to automatically check the morphologies of these elements. The proposed SSS-Net uses the sprint convolutional block (SCB), which uses asymmetric kernel convolutions in combination with depth-wise separable convolutions to reduce the overall cost of the network. SSS-Net is a shallow architecture with dense feature aggregation, which helps in better segmentation. The proposed SSS-Net consumes a smaller number of trainable parameters (4.04 million) compared to state-of-the-art methods. The SSS-Net was evaluated using a publicly available human blastocyst image dataset for component segmentation. The experimental results confirm that our proposal provides promising segmentation performance with a Jaccard Index of 82.88%, 77.40%, 88.39%, 84.94%, and 96.03% for ZP, TE, BL, ICM, and background, with residual connectivity, respectively. It is also provides a Jaccard Index of 84.51%, 78.15%, 88.68%, 84.50%, and 95.82% for ZP, TE, BL, ICM, and background, with dense connectivity, respectively. The proposed SSS-Net is providing a mean Jaccard Index (Mean JI) of 85.93% and 86.34% with residual and dense connectivity, respectively; this shows effective segmentation of blastocyst components for embryological analysis.

One-Bit Compressive Sensing: Can We Go Deep and Blind?

One-bit compressive sensing is concerned with the accurate recovery of an underlying sparse signal of interest from its one-bit noisy measurements. The conventional signal recovery approaches for this problem are mainly developed based on the assumption that an exact knowledge of the sensing matrix is available. In this work, however, we present a novel data-driven and model-based methodology that achieves <i>blind</i> recovery; i.e., signal recovery without requiring the knowledge of the sensing matrix. To this end, we make use of the deep unfolding technique and develop a model-driven deep neural architecture which is designed for this specific task. The proposed deep architecture is able to learn an alternative sensing matrix by taking advantage of the underlying unfolded algorithm such that the resulting <i>learned</i> recovery algorithm can accurately and quickly (in terms of the number of iterations) recover the underlying compressed signal of interest from its one-bit noisy measurements. In addition, due to the incorporation of the domain knowledge and the mathematical model of the system into the proposed deep architecture, the resulting network benefits from enhanced interpretability, has a very small number of trainable parameters, and requires very small number of training samples, as compared to the commonly used black-box deep neural network alternatives for the problem at hand.

NMF-DuNet: Nonnegative Matrix Factorization Inspired Deep Unrolling Networks for Hyperspectral and Multispectral Image Fusion

The fusion of high-resolution multispectral (HrMSI) and low-resolution hyperspectral images (LrHSI) has been acknowledged as a promising method for generating a high-resolution hyperspectral image (HrHSI), which is also termed to be an essential part for precise recognition and cataloguing of the underlying materials. In order to improve the fusion of LrHSI and HrMSI performance, in this article, we propose a novel Nonnegative Matrix Factorization Inspired Deep Unrolling Networks, dubbed NMF-DuNet, for fusing LrHSI and HrMSI. For this aim, initially, a variational fusion model regularized by non-negative sparse prior is proposed and then is solved through the gradient descent optimization method and unrolled towards the deep network. The nonnegative coefficient matrices and orthogonal of the proposed transform coefficients constraints are both incorporated into the proposed method. Moreover, the fusion of HrMSI and LrHSI heavily depends on an imaging model that explains the degeneracy of HSI in the spectral and spatial regions. Practically, the imaging model is often unknown. The degradation model is represented implicitly via a proposed network, and both the degradation model and sparse priors are jointly optimized through the training process of the proposed network. Instead of being hand-crafted, all the parameters of NMF-DuNet are learned end-to-end. Compared to the previous state-of-the-art model-based and learning-based fusion approaches, the hardware-friendly proposed NMF-DuNet outperforms both the model-based and learning-based fusion approaches and requires a far smaller number of trainable parameters and storage space while preserving real-time performance.

CGC-NET: Aircraft Detection in Remote Sensing Images Based on Lightweight Convolutional Neural Network

In the past few years, aircraft detection in remote sensing (RS) images has been an important research hotspot, and it is very crucial in plenty of military applications. Based on the high computational cost of the model and numerous parameters, deep convolution neural networks-based algorithms have excellent performance in the aircraft detection task. However, it is still difficult to detect aircraft due to the complex background of RS images, various types of aircraft, and so on. In addition, it is difficult and costly to make labels for satellite-based optical RS images. Consequently, we propose an end-to-end lightweight aircraft detection framework called CGC-NET (a network based on circle grayscale characteristics), which can accurately detect aircraft with a few training samples. There are only a small number of trainable parameters in CGC-NET, which greatly reduces the need for large datasets. Extensive evaluations indicate the excellent performance of CGC-NET, in which the <i>F</i>-score can reach 91.06&#x0025; and the model size is only 0.88 M. Therefore, CGC-NET can be used to accurately detect aircraft targets simply and effectively.