Hand-crafted Methods Research Articles

Polymer Informatics at Scale with Multitask Graph Neural Networks.

Artificial intelligence-based methods are becoming increasingly effective at screening libraries of polymers down to a selection that is manageable for experimental inquiry. The vast majority of presently adopted approaches for polymer screening rely on handcrafted chemostructural features extracted from polymer repeat units-a burdensome task as polymer libraries, which approximate the polymer chemical search space, progressively grow over time. Here, we demonstrate that directly "machine learning" important features from a polymer repeat unit is a cheap and viable alternative to extracting expensive features by hand. Our approach-based on graph neural networks, multitask learning, and other advanced deep learning techniques-speeds up feature extraction by 1-2 orders of magnitude relative to presently adopted handcrafted methods without compromising model accuracy for a variety of polymer property prediction tasks. We anticipate that our approach, which unlocks the screening of truly massive polymer libraries at scale, will enable more sophisticated and large scale screening technologies in the field of polymer informatics.

A feature boosted deep learning method for automatic facial expression recognition.

Automatic facial expression recognition (FER) plays a crucial role in human-computer based applications such as psychiatric treatment, classroom assessment, surveillance systems, and many others. However, automatic FER is challenging in real-time environment. The traditional methods used handcrafted methods for FER but mostly failed to produce superior results in the wild environment. In this regard, a deep learning-based FER approach with minimal parameters is proposed, which gives better results for lab-controlled and wild datasets. The method uses features boosting module with skip connections which help to focus on expression-specific features. The proposed approach is applied to FER-2013 (wild dataset), JAFFE (lab-controlled), and CK+ (lab-controlled) datasets which achieve accuracy of 70.21%, 96.16%, and 96.52%. The observed experimental results demonstrate that the proposed method outperforms the other related research concerning accuracy and time.

From Renaissance medals to the Jaguar E-Type car bonnet: mechanised production and the making of luxury goods

It is often assumed that the development of mechanised production technology and the making of luxury goods are incompatible. Arising from familiar narratives of the Industrial Revolution and echoes of Arts and Crafts thinking, high volume commodities are seen as the obvious domain for advances in mechanised production and high-quality luxury or semi-luxury goods are associated with traditional hand-craft methods. But innovations in mechanised production have sometimes been driven by the manufacture of luxury goods, not commodities. The technology which led to the pressing of the Jaguar E-Type bonnet began with the stamping of Renaissance medals, prestigious objects to bestow the favour of popes and princes. The major eighteenth-century advance in the making of three-dimensional objects in sheet metal from stamped components, a direct ancestor of twentieth-century car body manufacture, came from the Sheffield manufacture of luxury candlesticks and tableware, using silver-plated and silver sheet metal.

Mel spectrogram-based audio forgery detection using CNN

In this time of technology, digital speech can be created and falsified by a very diverse of hardware and software technologies. Audio copy-move forgery is an audio forgery technique that goals to create forged audio by hiding undesirable words or repeating wanted words in identical speech. Therefore, audio authentication has been a necessary requisition. In this study, an effective approach to spectral images based on audio copy-move forgery detection using convolutional neural networks (CNN) with data augmentation is proposed. There are only a few handcrafted methods conducted for the detection of audio copy-move forgery. None of the existing works on audio copy-move forgery detection has proposed deep feature learning from speech recording with Mel spectrogram. This is the first method to employ deep learning with Mel spectrogram of audio for the detection of audio copy-move forgery. The proposed CNN architecture classifies the suspicious Mel spectrogram images into two classes: original and forged. The proposed CNN system is successfully trained on these Mel spectrogram image feature extraction. The proposed algorithm has been tested on our datasets generated from Arabic Speech Corpus and TIMIT speech database. The results show the effectiveness, robustness of post-processing operations, and high accuracy of the proposed approach compared to other studies.

PHOTOGRAMMETRY NOW AND THEN – FROM HAND-CRAFTED TO DEEP-LEARNING TIE POINTS –

Abstract. Historical images provide a valuable source of information exploited by several kinds of applications, such as the monitoring of cities and territories, the reconstruction of destroyed buildings, and are increasingly being shared for cultural promotion projects through virtual reality or augmented reality applications. Finding reliable and accurate matches between historical and present images is a fundamental step for such tasks since they require to co-register the present 3D scene with the past one. Classical image matching solutions are sensitive to strong radiometric variations within the images, which are particularly relevant in these multi-temporal contexts due to different types of sensitive media (film/sensors) employed for the image acquisitions, different lighting conditions and viewpoint angles. In this work, we investigate the actual improvement provided by recent deep learning approaches to match historical and nowadays images. As learning-based methods have been trained to find reliable matches in challenging scenarios, including large viewpoint and illumination changes, they could overcome the limitations of classic hand-crafted methods such as SIFT and ORB. The most relevant approaches proposed by the research community in the last years are analyzed and compared using pairs of multi-temporal images.

X-ray image analysis for osteoporosis diagnosis: From shallow to deep analysis

Recently, automated disease diagnosis based on medical images has become an integral component of digital pathology packages. Texture analysis is commonly used to address this issue, particularly in the context of estimating the osteoporosis progression in bone samples. Most research in this context uses handcrafted methods to directly extract bones image features despite the substantial correlation between sick and healthy bones, which explains the limited results. In this work, the handcrafted feature extraction method (e.g. HOG and/or LPQ) will be applied to a set of descriptors obtained from a deep analysis of bone texture images using Gabor's filter bank. In addition, the classifier automatically adjusts the Gabor filters settings, using the bat-inspired algorithm based optimization, to achieve deep analysis behavior and optimal performance. Using a typically osteoporosis database, our experimental results reveal a significant improvement over the state-of-the-art deep/handcrafted techniques, resulting in an excellent performance of 89.66% for osteoporosis diagnosis.

An efficient stacked ensemble model for the detection of COVID-19 and skin cancer using fused feature of transfer learning and handcrafted methods

ABSTRACT Recently, COVID-19 and Skin cancer have been the most common diseases. If they haven’t been treated early, a severe illness may cause in the patients and may lead to death. Several automated detection methods have been explored in the area, but medical sectors need a high-performance CAD system. This work proposed a fused feature extraction technique containing a transfer learning of the DenseNet-169 model and six handcrafted methods to capture richer and more detailed features. Moreover, we propose a unique first-level stacked ensemble technique that considers RF, GBM, LR, SVM, and KNN algorithms with base-learner and meta-learner phases to enhance the performance of the classifier model. We have used two publicly available datasets for evaluation: SARS-CoV-2 CT Scan and ISIC Archive datasets. The result shows that all five implemented algorithms improve their performance in the meta-learner phase on both dataset types. For the SARS-CoV-2 CT Scan dataset, the LR performs better than the others in all evaluation metrics; it scores 99.73% accuracy. In contrast, RF is the highest performer for the ISIC Archive dataset, with 89.09% of accuracy. The comparison shows our proposed method outperforms the previously worked methods by most of the evaluation metrics in both datasets.

858P Automated detection of melanoma: Comparing a convolutional neural network (CNN) approach with an algorithm assessing disorder in the pattern of pigmented lesions, intended to mimick onco-dermatologists' visual analysis

Among Computer Aided Diagnosis (CAD) systems for melanoma detection, CNN behaves as a “black box”, using a huge number of unknown features from a lesion’s image to come to a classification and where no expert input can be implemented. Conversely of hand-crafted methods, the expert is selecting features to be used for the lesions' classification. Most of these hand-crafted algorithms were based on ABCD criteria (asymmetry, border, color, diameter). However the unconscious process leading onco-dermatologists to arbitrate between nevi and melanoma is more likely based on a global cognitive assessment, which we hypothezise to be an overall perception of order/disorder in the pattern of pigmented lesions. Our objective was to design an hand-crafted model based on the concept of ordered/disordered pattern, and to assess its performances by comparison to a CNN model, for the classification melanoma versus nevus. We trained and tested 2 algorithms on a dermoscopic images dataset of 1533 melanomas and 6124 nevi. First, we developped The "Disorder model" based on a hand-crafted method, we extracted 4 specific features to characterize melanoma disorder named entropy, skewness, standard deviation and kurtosis, from 4 colors spaces. The classification of melanoma versus nevi was based on a statistical clustering method (KNN algorithm). Second, we developped a CNN model on the same dataset. Performances of the CNN model yielded: AUC 0.89, sensitivity 86%, specificity 75%, balanced accuracy 81%. The “Disorder model” reached similar high performances: AUC 0.91, sensitivity 91%, specificity 74%, balanced accuracy 82.5%. A statistical χ2 test revealed that entropy, characterizing color disorder, was the most discriminative feature for melanoma detection. This study shows that an algorithm attempting to mimick human brain ability, using a few features to assess the disorder pattern in pigmented lesions, can reach a level of performance for melanoma detection equivalent to CNN-based algorithm, which is using much more features. Entropy seems to be a key feature to assess pattern disorder of pigmented lesions.

Feature Extraction with Handcrafted Methods and Convolutional Neural Networks for Facial Emotion Recognition

This research compares the facial expression recognition accuracy achieved using image features extracted (a) manually through handcrafted methods and (b) automatically through convolutional neural networks (CNNs) from different depths, with and without retraining. The Karolinska Directed Emotional Faces, Japanese Female Facial Expression, and Radboud Faces Database databases have been used, which differ in image number and characteristics. Local binary patterns and histogram of oriented gradients have been selected as handcrafted methods and the features extracted are examined in terms of image and cell size. Five CNNs have been used, including three from the residual architecture of increasing depth, Inception_v3, and EfficientNet-B0. The CNN-based features are extracted from the pre-trained networks from the 25%, 50%, 75%, and 100% of their depths and, after their retraining on the new databases. Each method is also evaluated in terms of calculation time. CNN-based feature extraction has proved to be more efficient since the classification results are superior and the computational time is shorter. The best performance is achieved when the features are extracted from shallower layers of pre-trained CNNs (50% or 75% of their depth), achieving high accuracy results with shorter computational time. CNN retraining is, in principle, beneficial in terms of classification accuracy, mainly for the larger databases by an average of 8%, also increasing the computational time by an average of 70%. Its contribution in terms of classification accuracy is minimal when applied in smaller databases. Finally, the effect of two types of noise on the models is examined, with ResNet50 appearing to be the most robust to noise.

ULMR: An Unsupervised Learning Framework for Mismatch Removal.

Due to radiometric and geometric distortions between images, mismatches are inevitable. Thus, a mismatch removal process is required for improving matching accuracy. Although deep learning methods have been proved to outperform handcraft methods in specific scenarios, including image identification and point cloud classification, most learning methods are supervised and are susceptible to incorrect labeling, and labeling data is a time-consuming task. This paper takes advantage of deep reinforcement leaning (DRL) and proposes a framework named unsupervised learning for mismatch removal (ULMR). Resorting to DRL, ULMR firstly scores each state–action pair guided by the output of classification network; then, it calculates the policy gradient of the expected reward; finally, through maximizing the expected reward of state–action pairings, the optimal network can be obtained. Compared to supervised learning methods (e.g., NM-Net and LFGC), unsupervised learning methods (e.g., ULCM), and handcraft methods (e.g., RANSAC, GMS), ULMR can obtain higher precision, more remaining correct matches, and fewer remaining false matches in testing experiments. Moreover, ULMR shows greater stability, better accuracy, and higher quality in application experiments, demonstrating reduced sampling times and higher compatibility with other classification networks in ablation experiments, indicating its great potential for further use.

Binary Neural Networks for Memory-Efficient and Effective Visual Place Recognition in Changing Environments

Visual place recognition (VPR) is a robot's ability to determine whether a place was visited before using visual data. While conventional hand-crafted methods for VPR fail under extreme environmental appearance changes, those based on convolutional neural networks (CNNs) achieve state-of-the-art performance but result in heavy runtime processes and model sizes that demand a large amount of memory. Hence, CNN-based approaches are unsuitable for resource-constrained platforms, such as small robots and drones. In this paper, we take a multi-step approach of decreasing the precision of model parameters, combining it with network depth reduction and fewer neurons in the classifier stage to propose a new class of highly compact models that drastically reduces the memory requirements and computational effort while maintaining state-of-the-art VPR performance. To the best of our knowledge, this is the first attempt to propose binary neural networks for solving the visual place recognition problem effectively under changing conditions and with significantly reduced resource requirements. Our best-performing binary neural network, dubbed FloppyNet, achieves comparable VPR performance when considered against its full-precision and deeper counterparts while consuming 99% less memory and increasing the inference speed seven times.

Vowel speech recognition from rat electroencephalography using long short-term memory neural network.

Over the years, considerable research has been conducted to investigate the mechanisms of speech perception and recognition. Electroencephalography (EEG) is a powerful tool for identifying brain activity; therefore, it has been widely used to determine the neural basis of speech recognition. In particular, for the classification of speech recognition, deep learning-based approaches are in the spotlight because they can automatically learn and extract representative features through end-to-end learning. This study aimed to identify particular components that are potentially related to phoneme representation in the rat brain and to discriminate brain activity for each vowel stimulus on a single-trial basis using a bidirectional long short-term memory (BiLSTM) network and classical machine learning methods. Nineteen male Sprague-Dawley rats subjected to microelectrode implantation surgery to record EEG signals from the bilateral anterior auditory fields were used. Five different vowel speech stimuli were chosen, /a/, /e/, /i/, /o/, and /u/, which have highly different formant frequencies. EEG recorded under randomly given vowel stimuli was minimally preprocessed and normalized by a z-score transformation to be used as input for the classification of speech recognition. The BiLSTM network showed the best performance among the classifiers by achieving an overall accuracy, f1-score, and Cohen’s κ values of 75.18%, 0.75, and 0.68, respectively, using a 10-fold cross-validation approach. These results indicate that LSTM layers can effectively model sequential data, such as EEG; hence, informative features can be derived through BiLSTM trained with end-to-end learning without any additional hand-crafted feature extraction methods.

Sustainable Oil Palm Resource Assessment Based on an Enhanced Deep Learning Method

Knowledge of the number and distribution of oil palm trees during the crop cycle is vital for sustainable management and predicting yields. The accuracy of the conventional image processing method is limited for the hand-crafted feature extraction method and the overfitting problem occurs due to the insufficient dataset. We propose a modification of the Faster Region-based Convolutional Neural Network (FRCNN) for palm tree detection to reduce the overfitting problem and improve the detection accuracy. The enhanced FRCNN (EFRCNN) leads to improved performance for detecting objects (in the same image) when they are of multiple sizes by using a feature concatenation method. Transfer learning based on a ResNet50 model is used to extract the features of the input image. High-resolution images of oil palm trees from a drone are used to form the data set, containing mature, young, and mixed oil palm tree regions. We train and test the EFRCNN, the FRCNN, a CNN used recently for oil palm image detection, and two standard methods, namely, the support vector machine (SVM) and template matching (TM). The results reveal an overall accuracy of ≥96.8% for the EFRCNN on the three test sets. The accuracy is higher than the CNN and FRCNN and substantially higher than SVM and TM. For large-scale plantations, the accuracy improvement is significant. This research provides a method for automatically counting the oil palm trees in large-scale plantations.

Use of deep learning-based radiomics to differentiate Parkinson's disease patients from normal controls: a study based on [18F]FDG PET imaging.

We proposed a novel deep learning-based radiomics (DLR) model to diagnose Parkinson's disease (PD) based on [18F]fluorodeoxyglucose (FDG) PET images. In this two-center study, 255 normal controls (NCs) and 103 PD patients were enrolled from Huashan Hospital, China; 26 NCs and 22 PD patients were enrolled as a separate test group from Wuxi 904 Hospital, China. The proposed DLR model consisted of a convolutional neural network-based feature encoder and a support vector machine (SVM) model-based classifier. The DLR model was trained and validated in the Huashan cohort and tested in the Wuxi cohort, and accuracy, sensitivity, specificity and receiver operator characteristic (ROC) curve graphs were used to describe the model's performance. Comparative experiments were performed based on four other models including the scale model, radiomics model, standard uptake value ratio (SUVR) model and DLR model. The DLR model demonstrated superiority in differentiating PD patients and NCs in comparison to other models, with an accuracy of 95.17% [90.35%, 98.13%] (95% confidence intervals, CI) in the Huashan cohort. Moreover, the DLR model also demonstrated greater performance in diagnosing PD early than routine methods, with an accuracy of 85.58% [78.60%, 91.57%] in the Huashan cohort. We developed a DLR model based on [18F]FDG PET images that showed good performance in the noninvasive, individualized prediction of PD and was superior to traditional handcrafted methods. This model has the potential to guide and facilitate clinical diagnosis and contribute to the development of precision treatment. The DLR method on [18F]FDG PET images helps clinicians to diagnose PD and PD subgroups from normal controls. A prospective two-center study showed that the DLR method provides greater diagnostic accuracy.

Micro-Expression Recognition Based on Optical Flow and PCANet.

Micro-expressions are rapid and subtle facial movements. Different from ordinary facial expressions in our daily life, micro-expressions are very difficult to detect and recognize. In recent years, due to a wide range of potential applications in many domains, micro-expression recognition has aroused extensive attention from computer vision. Because available micro-expression datasets are very small, deep neural network models with a huge number of parameters are prone to over-fitting. In this article, we propose an OF-PCANet+ method for micro-expression recognition, in which we design a spatiotemporal feature learning strategy based on shallow PCANet+ model, and we incorporate optical flow sequence stacking with the PCANet+ network to learn discriminative spatiotemporal features. We conduct comprehensive experiments on publicly available SMIC and CASME2 datasets. The results show that our lightweight model obviously outperforms popular hand-crafted methods and also achieves comparable performances with deep learning based methods, such as 3D-FCNN and ELRCN.

AERIAL TRIANGULATION WITH LEARNING-BASED TIE POINTS

Abstract. Aerial triangulation (AT) has reached outstanding progress in the last decades, and now fully automated solutions for nadir and oblique images are available. Usually, image correspondences (tie points) are found using hand-crafted methods, such as SIFT or its variants. But in the last years, there were many investigations and developments to promote the use of machine and deep learning solutions within the photogrammetric processing pipeline. The paper explores learning-based methods for the extraction of tie points in aerial image blocks. Image correspondences are used to perform aerial triangulation (AT) and successively generate dense point clouds. Two different datasets are used to compare conventional hand-crafted detector/descriptor methods with respect to learning-based methods. Accuracy analyses are performed using GCPs as well as ground truth LiDAR point clouds. Results confirm the potential of learningbased methods in finding reliable image correspondences in the aerial block, still showing space for improvements due to camera rotations.

A computer-aided diagnosis system for detecting various diabetic retinopathy grades based on a hybrid deep learning technique

Diabetic retinopathy (DR) is a serious disease that may cause vision loss unawares without any alarm. Therefore, it is essential to scan and audit the DR progress continuously. In this respect, deep learning techniques achieved great success in medical image analysis. Deep convolution neural network (CNN) architectures are widely used in multi-label (ML) classification. It helps in diagnosing normal and various DR grades: mild, moderate, and severe non-proliferative DR (NPDR) and proliferative DR (PDR). DR grades are formulated by appearing multiple DR lesions simultaneously on the color retinal fundus images. Many lesion types have various features that are difficult to segment and distinguished by utilizing conventional and hand-crafted methods. Therefore, the practical solution is to utilize an effective CNN model. In this paper, we present a novel hybrid, deep learning technique, which is called E-DenseNet. We integrated EyeNet and DenseNet models based on transfer learning. We customized the traditional EyeNet by inserting the dense blocks and optimized the resulting hybrid E-DensNet model’s hyperparameters. The proposed system based on the E-DenseNet model can accurately diagnose healthy and different DR grades from various small and large ML color fundus images. We trained and tested our model on four different datasets that were published from 2006 to 2019. The proposed system achieved an average accuracy (ACC), sensitivity (SEN), specificity (SPE), Dice similarity coefficient (DSC), the quadratic Kappa score (QKS), and the calculation time (T) in minutes (m) equal 91.2%, 96%, 69%, 92.45%, 0.883, and 3.5m respectively. The experiments show promising results as compared with other systems.Graphical abstract

Auxiliary classification of cervical cells based on multi-domain hybrid deep learning framework

Computer-aided cervical cell classification using Pap smears or Thinprep cytologic test (TCT) have been widely applied as a high effective screening tool, by which the cells are classified into different subclasses. However, existing classification approaches mainly rely on single detection structure, like deep learning or hand-crafted methods, which have huge computation complexity and lower accuracy. So far, no cell spectrum is applied for classification. This paper addresses the limitations by making the first attempt to use the multi-domain hybrid deep learning framework (MDHDN) for the classification of cervical cells. Cell deep features from multi-domain (time and frequency) are extracted by the pretrained VGG-19 (Visual Geometry Group-19), which is the deep Convolutional Neural Network (CNN) with a hashing layer after the last fully connected layer. Hand-crafted features for the original images are processed with the feature selection, clustering and dimensionality reduction. Then the three subchannels of the proposed framework output the category results using the SVM classifier, the final cell diagnosis is generated by the correlation analysis. Results show that the proposed approach obtains the similar performance with the state-of-the-art models using the novel structure, whose accuracy, sensitivity, and specificity are 98.7%, 98.2%, 98.9% in Herlev dataset when applying five-fold cross-validation, respectively. Similar superior classification performance is achieved on the BJTU dataset, validation on the SIPaKMeD dataset also proves its generalization ability. The proposed novel screening framework is promising for the early diagnosis of cervical cancer, multi-domain and hybrid features are proved feasible in clinical practice.

AI Based COVID Pneumonia Classifier Using Machine Learning

To classify and detect the presence of pneumonia or COVID-19 in a series of chest X-ray images, this research presents a convolutional neural network model that was trained from the ground up. Many other methods use transfer learning or handcrafted methods to attain a high level of accuracy in classifying data. Pneumonia or COVID-19 can be diagnosed using a convolutional neural network model that extracts information from the provided chest X-ray image and classifies it. The reliability and interpretability issues that arise frequently when dealing with medical images may be alleviated by using this method. Pneumonia datasets are scarce for this classification assignment, as they are for other deep learning classification tasks that require a huge image library. To increase the CNN model's validation and classification accuracy, we applied a variety of data augmentation strategies. We were able to achieve outstanding validation accuracy.

Restorable-inpainting: A novel deep learning approach for shoeprint restoration

Shoeprints are important information collected at the crime scene and are of great value for forensic analysis. Shoeprints collected in real-world scenarios are normally unclear, abrasive, and lack contextual and other kinds of missing information. In this research, we apply a novel deep learning technique called restorable inpainting to repair shoeprint contours and missing parts. Existing inpainting methods aim to fill artificially occluded areas with plausible pixels, but these methods may not restore missing information for occlusions in shoeprint images. In addition, because no ground-truth shoeprints exist for training samples, inpainting occluded regions becomes challenging. In this paper, we propose DeepShoePaint, a novel deep learning approach to perform restorable inpainting by restoring synthetic information resembling desirable shoeprint images necessary for forensics. DeepShoePaint novelly adapts a probabilistic distribution borrowed from the variational autoencoder into a U-Net-like structure forming a unified architecture trained in an unsupervised fashion to restore occluded and masked regions to produce human-verifiable shoeprints. The experimental results reveal that DeepShoePaint achieves outstanding human inspection and statistical assessment results and outperforms conventional inpainting models. We believe that this study can provide valuable insights, not limited to inpainting, into restoring desirable shoeprints to automate and facilitate the forensic investigation and examination process instead of using handcrafted methods.