Deep Pre-trained Models Research Articles

NFT price and sales characteristics prediction by transfer learning of visual attributes

Non-fungible tokens (NFTs) are unique digital assets whose possession is defined over a blockchain. NFTs can represent multiple distinct objects such as art, images, videos, etc. There was a recent surge of interest in trading them which makes them another type of alternative investment. The inherent volatility of NFT prices, attributed to factors such as over-speculation, liquidity constraints, rarity, and market volatility, presents challenges for accurate price predictions. For such analysis and forecasting, machine learning methods offer a robust solution framework.Here, we focus on three related prediction problems over NFTs: Predicting NFTs sale price, inferring whether a given NFT will participate in a secondary sale, and predicting NFT's sale price change over time. We analyze and learn the visual characteristics of NFTs by deep pre-trained models and combine such visual knowledge with additional important non-visual attributes such as the sale history, seller's and buyer's centralities in the trading network, and collection's resale probability. We categorize input NFTs into six categories based on their characteristics. Across detailed experiments, we found visual attributes obtained from deep pre-trained models to increase the prediction performance in all cases, and EfficientNet seems to perform the best. In general, CNN and XGBoost consistently outperformed the rest of them across all categories. We also publish our novel NFT dataset with temporal price knowledge, which is the first dataset to have NFT prices over time rather than at a single time point. Our code and NFT datasets are publicly available at https://github.com/seferlab/deep_nft.

Abnormal Heart Sound Classification and Model Interpretability: A Transfer Learning Approach with Deep Learning

Physician detection of heart sound abnormality is complicated by the inherent difficulty of detecting critical abnormalities in the presence of noise. Computer-aided heart auscultation provides a promising alternative for more accurate detection, with recent deep learning approaches exceeding expert accuracy. Although combining phonocardiogram (PCG) data with electrocardiogram (ECG) data provides more information to an abnormal heart sound classifier, the scarce presence of labelled datasets with this combination impedes training. This paper explores fine-tuning deep convolutional neural networks such as ResNet, VGG, and inceptionv3, on images of spectrograms, mel-spectrograms, and scalograms. By fine-tuning deep pre-trained models on image representations of ECG and PCG, we achieve 91.25% accuracy on the training-a dataset of the PhysioNet Computing in Cardiology Challenge 2016, compared to a previous result of 81.48%. Interpretation of the model’s learned features is also provided, with the results indicative of clinical significance.

Cascade Graph Neural Networks for Few-Shot Learning on Point Clouds

Point cloud data, a flexible 3D object representation, is critical for various applications such as autonomous driving, robotics and remote sensing. Despite the recent success of deep neural networks (DNNs) on supervised point cloud analysis tasks, they still rely on tedious manual annotation of point clouds and cannot make predictions for new classes. Unlike few-shot learning for 2D images with the advantages of large-scale datasets and high-quality deep pre-trained models like ResNet, for 3D few-shot learning, obtaining discriminative representations of unseen classes with high intra-class similarity and inter-class difference is very challenging. To address this issue, this work proposes a novel cascade graph neural network for few-shot learning on point clouds, termed as CGNN, in which two cascade GNNs are adopted to extract the intra-object topological information and learn the inter-object relations respectively. To further increase the discriminability of point cloud features, we first design a novel discriminative edge label to model the intra-class similarity and inter-class dissimilarity based on channel-wise feature variance and class consistency. Second, we propose a novel few-shot circle loss which classifies the nodes into two subsets, i.e., support to support pairs and support to query pairs, and optimizes the pair-wise similarity on two subsets independently. Extensive experiments on benchmark CAD and real LiDAR point cloud datasets have demonstrated that CGNN improves accuracy by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.98\%$</tex-math> </inline-formula> over the state-of-the-art GNN-based few-shot classification methods.

From Big to Small: Adaptive Learning to Partial-Set Domains.

Domain adaptation targets at knowledge acquisition and dissemination from a labeled source domain to an unlabeled target domain under distribution shift. Still, the common requirement of identical class space shared across domains hinders applications of domain adaptation to partial-set domains. Recent advances show that deep pre-trained models of large scale endow rich knowledge to tackle diverse downstream tasks of small scale. Thus, there is a strong incentive to adapt models from large-scale domains to small-scale domains. This paper introduces Partial Domain Adaptation (PDA), a learning paradigm that relaxes the identical class space assumption to that the source class space subsumes the target class space. First, we present a theoretical analysis of partial domain adaptation, which uncovers the importance of estimating the transferable probability of each class and each instance across domains. Then, we propose Selective Adversarial Network (SAN and SAN++) with a bi-level selection strategy and an adversarial adaptation mechanism. The bi-level selection strategy up-weighs each class and each instance simultaneously for source supervised training, target self-training, and source-target adversarial adaptation through the transferable probability estimated alternately by the model. Experiments on standard partial-set datasets and more challenging tasks with superclasses show that SAN++ outperforms several domain adaptation methods.

Pipeline Irregular Defect Inversion for Magnetic Flux Leakage Detection System Based on Heterogeneous Multiclass Feature Fusion

Defect inversion estimates the defect size quantitatively, which is a significant process in magnetic flux leakage (MFL) detection system. The actual measured defects are irregular due to the complex environment, which leads to signal deformation. Moreover, the sufficient and accurate features of irregular defects are hard to extract, so it is difficult to invert MFL irregular defects precisely. Therefore, an end-to-end irregular defect inversion method based on heterogeneous multi-class feature fusion (HMFF) is proposed to quantify pipeline defects for MFL detection system. The HMFF extracts the abstract features and external features of MFL heterogeneous signals, respectively, where the abstract features are extracted by two deep pre-trained models with the same structure, and the external features are extracted with guidance from expert experience. Then, two feature fusion stages are proposed to fuse the extracted multi-class features from MFL heterogeneous signals for the first time. Among them, multi-class feature fusion not only makes full use of deep abstract features, but also enhances the supervision of expert experience to improve the interpretability. In addition, the heterogeneous feature fusion enriches the extracted features, and reduces the impact of signal deformation. Finally, the fully connected layers with appropriate activation functions are applied to invert the defect sizes. The experiments on MFL measured signals are conducted to demonstrate the precision and robustness of the proposed method.

Dynamical Fusion Model With Joint Variational and Deep Priors for Hyperspectral Image Super-Resolution

In this paper, we propose a novel dynamic fusion model (DFM) with joint variational and deep priors for the task of hyperspectral image super-resolution (HISR). The given model can benefit from both the advantages of traditional modeling and deep learning methods, thus achieving significant improvements based on existing deep pre-trained models. Specifically, the given model mainly contains two new designed terms, i.e., the weighted spatial fidelity (WSF) term and the deep fusion (DF) term. The WSF term focuses on the spatial recovery of the low-resolution hyperspectral image through the high-resolution multispectral image without the knowledge of the spectral response matrix, thus the proposed DFM can be viewed as a semi-blind model for HISR. Moreover, the DF term relied upon deep fusion with a designed adaptive weight matrix, which can effectively inject the deep priors into the traditional minimization model. Besides, the proposed DFM can be quickly and effectively solved using the alternating direction method of multipliers. Experimental results on widely used datasets demonstrate the superiority of our approach compared with state-of-the-art HISR methods.

Fast 3D Face Reconstruction from a Single Image Using Different Deep Learning Approaches for Facial Palsy Patients.

The 3D reconstruction of an accurate face model is essential for delivering reliable feedback for clinical decision support. Medical imaging and specific depth sensors are accurate but not suitable for an easy-to-use and portable tool. The recent development of deep learning (DL) models opens new challenges for 3D shape reconstruction from a single image. However, the 3D face shape reconstruction of facial palsy patients is still a challenge, and this has not been investigated. The contribution of the present study is to apply these state-of-the-art methods to reconstruct the 3D face shape models of facial palsy patients in natural and mimic postures from one single image. Three different methods (3D Basel Morphable model and two 3D Deep Pre-trained models) were applied to the dataset of two healthy subjects and two facial palsy patients. The reconstructed outcomes were compared to the 3D shapes reconstructed using Kinect-driven and MRI-based information. As a result, the best mean error of the reconstructed face according to the Kinect-driven reconstructed shape is 1.5±1.1&amp;nbsp;mm. The best error range is 1.9±1.4&amp;nbsp;mm when compared to the MRI-based shapes. Before using the procedure to reconstruct the 3D faces of patients with facial palsy or other facial disorders, several ideas for increasing the accuracy of the reconstruction can be discussed based on the results. This present study opens new avenues for the fast reconstruction of the 3D face shapes of facial palsy patients from a single image. As perspectives, the best DL method will be implemented into our computer-aided decision support system for facial disorders.

Coronavirus Detection using Deep Pre-trained Model on Chest X-ray Imaging

The constant increase in the number of patients suffering from the pandemic Covid-19 has overpowered the medical management system over the world. This pandemic has affected every aspect of human life all over the world. However, the limited number of testing kits is making it more challenging to deal with the global pandemic. It is near to impossible for people to maintain social distancing all the time, and lengthy testing procedure is one of the major cause of the rapid spread of the epidemic Covid-19. The polymerase chain reaction test that is done in real-time (RT-PCR) is time-consuming for each patient with gasping disease to be tested. However, the use of a chest x-ray machine tool can accelerate testing time. X-ray machines are now accessible in almost every medical management healthcare system, with instant results. The proposed research aims to detect the pandemic. The dataset was collected from an open-source repository that consists of the chest x-ray images of patients who were Covid-19 positive, and covid-19 negative which means not infected by the pandemic coronavirus but suffering from normal pneumonia. The approach of this paper uses a deep convolutional neural network model i-e VGG16, to begin the process of classifying images into positive or negative. In the proposed research VGG16, the Deep Convolutional Network model is used, this is a pre-trained model, previously trained on the ImageNet dataset that com-prises 14M images that belong to 1K different classes. This model is fine-tuned on 1202 images. The preprocessing of collected data is done by converting it into an RGB channel and resizing it to match the requirement of our model i-e VGG16. Data augmentation is performed on the preprocessed dataset. Our proposed model gives 99.1%, 100%, and 98.59% accuracies, sensitivity specificity respectively for the detection of Covid-19 on patients’ cxr images.

A Novel MRI Diagnosis Method for Brain Tumor Classification Based on CNN and Bayesian Optimization.

Brain tumor is one of the most aggressive diseases nowadays, resulting in a very short life span if it is diagnosed at an advanced stage. The treatment planning phase is thus essential for enhancing the quality of life for patients. The use of Magnetic Resonance Imaging (MRI) in the diagnosis of brain tumors is extremely widespread, but the manual interpretation of large amounts of images requires considerable effort and is prone to human errors. Hence, an automated method is necessary to identify the most common brain tumors. Convolutional Neural Network (CNN) architectures are successful in image classification due to their high layer count, which enables them to conceive the features effectively on their own. The tuning of CNN hyperparameters is critical in every dataset since it has a significant impact on the efficiency of the training model. Given the high dimensionality and complexity of the data, manual hyperparameter tuning would take an inordinate amount of time, with the possibility of failing to identify the optimal hyperparameters. In this paper, we proposed a Bayesian Optimization-based efficient hyperparameter optimization technique for CNN. This method was evaluated by classifying 3064 T-1-weighted CE-MRI images into three types of brain tumors (Glioma, Meningioma, and Pituitary). Based on Transfer Learning, the performance of five well-recognized deep pre-trained models is compared with that of the optimized CNN. After using Bayesian Optimization, our CNN was able to attain 98.70% validation accuracy at best without data augmentation or cropping lesion techniques, while VGG16, VGG19, ResNet50, InceptionV3, and DenseNet201 achieved 97.08%, 96.43%, 89.29%, 92.86%, and 94.81% validation accuracy, respectively. Moreover, the proposed model outperforms state-of-the-art methods on the CE-MRI dataset, demonstrating the feasibility of automating hyperparameter optimization.

MacularNet: Towards Fully Automated Attention-Based Deep CNN for Macular Disease Classification

In this work, we propose an attention-based deep convolutional neural network (CNN) model as an assistive computer-aided tool to classify common types of macular diseases: age-related macular degeneration, diabetic macular edema, diabetic retinopathy, choroidal neovascularization, macular hole, and central serous retinopathy from normal macular conditions with the help of scans from optical coherence tomography (OCT) imaging. Our proposed architecture unifies refined deep pre-trained models using transfer learning with limited training data and a deformation-aware attention mechanism encoding crucial morphological variations appearing in the deformation of retinal layers, detachments from the subsequent layers, presence of fluid-filled regions, geographic atrophy, scars, cysts, drusen, to achieve superior macular imaging classification performance. The proposed attention module facilitates the base network to automatically focus on the salient features arising due to the macular structural abnormalities while suppressing the irrelevant (or no cues) regions. The superiority of our proposed method lies in the fact that it does not require any pre-processing steps such as retinal flattening, denoising, and selection of a region of interest making it fully automatic and end-to-end trainable. Additionally, it requires a reduced number of network model parameters while achieving higher diagnostic performance. Extensive experimental results, analysis on four datasets along with the ablation studies show that the proposed architecture achieves state-of-the-art performance.

Deep transfer learning based model for colorectal cancer histopathology segmentation: A comparative study of deep pre-trained models

Colorectal cancer is one of the leading causes of cancer-related death, worldwide. Early detection of suspicious tissues can significantly improve the survival rate. In this study, the performance of a wide variety of deep learning-based architectures is evaluated for automatic tumor segmentation of colorectal tissue samples. The proposed approach highlights the utility of incorporating convolutional neural network modules and transfer learning in the encoder part of a segmentation architecture for histopathology image analysis. A comparative and extensive experiment was conducted on a challenging histopathological segmentation task to demonstrate the effectiveness of incorporating deep modules in the segmentation encoder-decoder network as well as the contributions of its components. Experimental results demonstrate that shared DenseNet and LinkNet architecture is promising, achieves the state-of-the-art performance, and outperforms other methods with a dice similarity index of 82.74%±1.77, accuracy of 87.07%±1.56, and f1-score value of 82.79%±1.79.

Application of Deep Transfer Learning to the Classification of Colorectal Cancer Lymph Node Metastasis

Abstract Accurate classifications of colorectal cancer (CRC) lymph node metastasis (LNM) could assist radiologists in increasing the diagnostic accuracy and help surgeons establish a correct surgical plan. This study aims to present an efficient pipeline with deep transfer learning for CRC LNM classification. Hence, 11 deep pre-trained models have been investigated on a CRC LN dataset. The dataset of this experiment is from Harbin Medical University Cancer Hospital. This dataset contains samples of 619 patients. Among these samples, 312 were positive and 307 were negative. In addition, datasets with different dimensions and various training epochs were also studied to ascertain the minimum training dataset and training times. In order to improve the interpretability of the model classification performance, a visual convolution layer feature map was first established to compute the similarity distance between the feature map and original data. The experimental results revealed that resnet_152 was the best deep pre-trained model for the classification of CRC LNM, with an accuracy of 97.2%, with 600 raw data samples being the minimum dimension of a dataset and 30 epochs the minimum training times in the CRC LNM classification. This study suggests that the proposed deep transfer learning pipeline could classify the CRC LNM with high efficiency, without requiring sophisticated computational knowledge for radiologists.

Deep Fusion Feature Extraction for Caries Detection on Dental Panoramic Radiographs

Caries is the most well-known disease and relates to the oral health of billions of people around the world. Despite the importance and necessity of a well-designed detection method, studies in caries detection are still limited and show a restriction in performance. In this paper, we proposed a computer-aided diagnosis (CAD) method to detect caries among normal patients using dental radiographs. The proposed method mainly consists of two processes: feature extraction and classification. In the feature extraction phase, the chosen 2D tooth image was employed to extract deep activated features using a deep pre-trained model and geometric features using mathematic formulas. Both feature sets were then combined, called fusion feature, to complement each other defects. Then, the optimal fusion feature set was fed into well-known classification models such as support vector machine (SVM), k-nearest neighbor (KNN), decision tree (DT), Naïve Bayes (NB), and random forest (RF) to determine the best classification model that fit the fusion features set and perform the most preeminent result. The results show 91.70%, 90.43%, and 92.67% for accuracy, sensitivity, and specificity, respectively. The proposed method has outperformed the previous state-of-the-art and shows promising results when none of the measured factors is less than 90%; therefore, the method is promising for dentists and capable of wide-scale implementation caries detection in hospitals.

MSENet: Multi-Modal Squeeze-and-Excitation Network for Brain Tumor Severity Prediction

Classification of brain tumors is one of the daunting tasks in medical imaging and incorrect decisions during the diagnosis process may lead to increased human fatality. Latest advances in artificial intelligence and deep learning have opened the path for the success of numerous medical image analysis tasks, including the recognition of brain tumors. In this paper, we propose a simple architecture based on deep learning that results in strong generalization without needing much preprocessing. The proposed Multi-modal Squeeze and Excitation model (MSENet) receives multiple representations of a given tumor image, learns end-to-end and effectively predicts the severity level of tumor. Convolution feature descriptors from multiple deep pre-trained models are used to effectively describe the tumor images and are supplied as input to the proposed MSENet. The squeeze and excitation blocks of the MSENet allow the model to prioritize tumor regions while giving less emphasis to the rest of the image, serving as an attention mechanism in the model. The proposed model is evaluated on the benchmark brain tumor dataset that is publicly accessible from the Figshare repository. Experimental studies reveal that in terms of model parameters, the proposed approach is simple and leads to competitive performance compared to those of existing complex models. By increasing complexity, the proposed model leads to generalize better and achieves state-of-the-art accuracy of 96.05% on Figshare dataset. Compared to the existing models, the proposed model neither uses segmentation nor uses augmentation techniques and without much pre-processing achieves competitive performance.

Applying Different Machine Learning Techniques for Prediction of COVID-19 Severity.

Due to the increase in the number of patients who died as a result of the SARS-CoV-2 virus around the world, researchers are working tirelessly to find technological solutions to help doctors in their daily work. Fast and accurate Artificial Intelligence (AI) techniques are needed to assist doctors in their decisions to predict the severity and mortality risk of a patient. Early prediction of patient severity would help in saving hospital resources and decrease the continual death of patients by providing early medication actions. Currently, X-ray images are used as early symptoms in detecting COVID-19 patients. Therefore, in this research, a prediction model has been built to predict different levels of severity risks for the COVID-19 patient based on X-ray images by applying machine learning techniques. To build the proposed model, CheXNet deep pre-trained model and hybrid handcrafted techniques were applied to extract features, two different methods: Principal Component Analysis (PCA) and Recursive Feature Elimination (RFE) were integrated to select the most important features, and then, six machine learning techniques were applied. For handcrafted features, the experiments proved that merging the features that have been selected by PCA and RFE together (PCA + RFE) achieved the best results with all classifiers compared with using all features or using the features selected by PCA or RFE individually. The XGBoost classifier achieved the best performance with the merged (PCA + RFE) features, where it accomplished 97% accuracy, 98% precision, 95% recall, 96% f1-score and 100% roc-auc. Also, SVM carried out the same results with some minor differences, but overall it was a good performance where it accomplished 97% accuracy, 96% precision, 95% recall, 95% f1-score and 99% roc-auc. On the other hand, for pre-trained CheXNet features, Extra Tree and SVM classifiers with RFE achieved 99.6% for all measures.

Arabic (Indian) digit handwritten recognition using recurrent transfer deep architecture

The rapid volume of digit texts and images motivates researchers to build solid and efficient prediction models to recognize such media. The Arabic language is considered one of the difficult languages regarding the way of writing characters and digits. Recent research focuses on such language for building predictive approaches to recognize written materials. The Arabic (Indian) digit recognition task has been a challenging task and has gained more attention from researchers who build optimal predictive models from historical images that are used in many applications. However, transfer learning approaches exploit deep pre-trained models that could be re-used for similar tasks. So, in this paper, we propose an adapted deep hybrid transfer model developed using two well-known pre-trained convolutional neural networks (CNN) models. These are further adapted by adding recurrent neural networks especially long short-term memory (LSTM) architectures to detect Arabic (Indian) Handwritten Digits (AHD). The CNN model learns the relevant features of Arabic (Indian) digits, while the sequence learning process in the LSTM layers extracts long-term dependence features. The experimental results, using popular datasets, show significant performance obtained by the adapted transfer models with accuracy reached up to 98.92% as well as with precision and recall values at most cases reached to 100% with statistical t test using p-value ( $$p\le 0.05$$ ) compared to baseline methods.

CCDF: Automatic system for segmentation and recognition of fruit crops diseases based on correlation coefficient and deep CNN features

In the agriculture farming business, plant diseases are the major reason for monetary misfortunes around the globe. It is an imperative factor, as it causes significant diminution in both quality and capacity of growing crops. Therefore, detection and taxonomy of various plants diseases is crucial, and it demands utmost attention. In plants, fruits act a major source of nutrients worldwide, however, various range of diseases adversely affect the production as well as the quality of the fruits. Therefore, utilization of an efficient machine vision technology not only detects the diseases at their early stages but also classify them accordingly. This research is primarily focusing on the detection and classification of various fruits diseases based on correlation coefficient and deep features (CCDF). The proposed technique incorporates two major steps of infected regions detection and finally feature extraction and classification. In the first step, initially contrast of input image is enhanced by utilizing a hybrid method - followed by proposed correlation coefficient-based segmentation method which separates the infected regions from the background. In the second step, two deep pre-trained models (VGG16, caffe AlexNet) are utilized for feature extraction of selected diseases (apple scab, apple rot, banana sigotka, banana cordial leaf spot, banana diamond leaf spot and deightoniella leaf and fruit spot). Parallel features fusion step is embedded to consolidate the extracted features prior to max-pooling step. Selection of most discriminant features are being performed using genetic algorithm before subjecting to the final stage of classification using mutli-class SVM. Experiments are being performed on publicly available datasets - plant village and CASC-IFW datasets to achieve the classification accuracy of 98.60%. Qualitative analysis of achieved results clearly shows that the proposed method outperforms several existing methods in terms of greater precision and improved classification accuracy.