Multimodal ensemble neural network system for skin cancer detection on heterogeneous dermatological data

Simple SummarySkin cancer is one of the most common cancers in humans. This study aims to create a system for recognizing pigmented skin lesions by analyzing heterogeneous data based on a multimodal neural network. Fusing patient statistics and multidimensional visual data allows for finding additional links between dermoscopic images and medical diagnostic results, significantly improving neural network classification accuracy. The use by specialists of the proposed system of neural network recognition of pigmented skin lesions will enhance the efficiency of diagnosis compared to visual diagnostic methods.Today, skin cancer is one of the most common malignant neoplasms in the human body. Diagnosis of pigmented lesions is challenging even for experienced dermatologists due to the wide range of morphological manifestations. Artificial intelligence technologies are capable of equaling and even surpassing the capabilities of a dermatologist in terms of efficiency. The main problem of implementing intellectual analysis systems is low accuracy. One of the possible ways to increase this indicator is using stages of preliminary processing of visual data and the use of heterogeneous data. The article proposes a multimodal neural network system for identifying pigmented skin lesions with a preliminary identification, and removing hair from dermatoscopic images. The novelty of the proposed system lies in the joint use of the stage of preliminary cleaning of hair structures and a multimodal neural network system for the analysis of heterogeneous data. The accuracy of pigmented skin lesions recognition in 10 diagnostically significant categories in the proposed system was 83.6%. The use of the proposed system by dermatologists as an auxiliary diagnostic method will minimize the impact of the human factor, assist in making medical decisions, and expand the possibilities of early detection of skin cancer.

Utilization of Image-Based Deep Learning in Multimodal Glaucoma Detection Neural Network from a Primary Patient Cohort.

Biologically inspired control system is necessary to be increased. This paper proposed the new design of multimodal neural network inspired from human learning system which takes different action in different condition. The multimodal neural network consists of some recurrent neural networks (RNNs) those are separated into different condition. There is selector system that decides certain RNN system depending the current condition of the robot. In this paper, we implemented this system in pendulum mobile robot as the basic object of study. Several certain number of RNNs are implemented into certain different condition of tilt robot. RNN works alternately depending on the condition of robot. In order to prove the effectiveness of the proposed model, we simulated in the computer simulation Open Dynamic Engine (ODE) and compared with ordinary RNN. The proposed neural model successfully stabilize the applied robot (2-wheeled robot). This model is developed for implemented into humanoid balancing learning system as the final object of study.

e13572 Background: Prediction of drug response is a critical research area in precision oncology and has been previously explored with large drug screening studies of cancer cell lines (CCLs). Patient-derived xenografts (PDXs) are an appealing platform for preclinical drug studies because the in vivo environment of PDXs helps preserve tumor heterogeneity and usually better mimics drug response of patients with cancer compared to CCLs. Methods: We investigate multimodal neural network (NN) and data augmentation for drug response prediction in PDXs. The multimodal NN learns to predict response using drug descriptors, gene expressions (GE), and histology whole-slide images (WSIs) where the multi-modality refers to tumor features only. The NN uses late integration where separate subnetworks are used to encode the input feature types before concatenation and prediction layers. Median tumor volume per treatment group is assessed relative to the control group to create a binary variable representing response. The data include twelve single-drug and 36 drug-pair treatments resulting in 2,556 single-drug and 2,203 drug-pair response values. Pathology and omics data from 487 PDXs from NCI's Patient Derived Models Repository are used as tumor feature model inputs. We explore whether the integration of WSIs with GE improves predictions as compared with models that use GE alone. We use two methods to address the limited number of response values in the dataset: 1) homogenize drug representations which allows to combine single-drug and drug-pairs into a single dataset, 2) augment drug-pair samples by switching the order of drug features which doubles the sample size of all drug-pair samples. These methods enable us to combine single-drug and drug-pair treatments which results in 6,962 responses, allowing us to train multimodal and unimodal NNs without changing architectures or the dataset. Results: Prediction performance of three unimodal NNs which use GE (um1, um2, and um3) are compared to assess the contribution of data augmentation methods. NN um1 that uses the full dataset which includes the original and the augmented drug-pair treatments as well as single-drug treatments significantly outperforms NNs (p-values < 0.01) that ignore either the augmented drug-pairs (um2) or the single-drug treatments (um3). In assessing the contribution of multimodal learning, results show that the multimodal NN (mm) outperforms both unimodal NNs that ignore either the GE (um4) or the WSIs (um1). However, the improvement of mm over um1 is not statistically significant (p-value < 0.26). Conclusions: Our results show that data augmentation and integration of histology images and GE can help improve prediction performance of drug response in PDXs.[Table: see text]

The study intended to explore the effect of different neural network algorithms in the electrocardiogram (ECG) classification of patients with congenital heart disease (CHD). Based on the single convolutional neural network (CNN) ECG algorithm and the recurrent neural network (RNN) ECG algorithm, a multimodal neural network (MNN) ECG algorithm was constructed utilizing the MIT-BIH database as training set and test set. Furthermore, the MNN ECG algorithm was optimized to establish an improved MNN (IMNN) algorithm, which was applied to the diagnosis of CHD patients. The CHD patients admitted between August 2016 and August 2019 were selected for analysis to compare the classification effect and accuracy rate of IMNN, MNN, CNN ECG, and RNN ECG algorithms. It was found that the RNN ECG algorithm had higher classification sensitivity and true positive rate in terms of normal or bundle (NB) branch block beat, supraventricular abnormal (SA) rhythm, abnormal ventricular (AV) beat, and fusion beat (FB) than the CNN ECG algorithm ( P < 0.05 ), and the classification sensitivity and true positive rate of IMNN algorithm in the four aspects were significantly higher than those of MNN algorithm ( P < 0.05 ). The classification accuracy of CNN ECG algorithm and RNN ECG algorithm was above 98%, while that of MNN algorithm and IMNN algorithm was better than that of CNN ECG algorithm and RNN ECG algorithm, and the accuracy rate can reach 98.5% or more. Moreover, the accuracy rate of the IMNN algorithm can reach more than 98%. In conclusion, IMNN not only has a good classification ability in the simulated environment but also performs well in the actual environment, which is worthy of clinical promotion.

Many engineered approaches have been proposed over the years for solving the hard problem of performing indoor localization using smartphone sensors. However, specialising these solutions for difficult edge cases remains challenging. Here we propose an end-to-end hybrid multimodal deep neural network localization system, MM-Loc, relying on zero hand-engineered features, but learning automatically from data instead. This is achieved by using modality-specific neural networks to extract preliminary features from each sensing modality, which are then combined by cross-modality neural structures. We show that our choice of modality-specific neural architectures can estimate the location independently. But for better accuracy, a multimodal neural network that fuses the features of early modality-specific representations is a better proposition. Our proposed MM-Loc system is tested on cross-modality samples characterised by different sampling rate and data representation (inertial sensors, magnetic and WiFi signals), outperforming traditional approaches for location estimation. MM-Loc elegantly trains directly from data unlike conventional indoor positioning systems, which rely on human intuition.

Intelligent skin cancer diagnosis using improved particle swarm optimization and deep learning models

We developed a multi-modal feed-forward neural network to predict the secondary structure of proteins. Several neural networks are used together and the final prediction results are decided by majority rule. We used 6137 residues to train and test the method. The average accuracy of the prediction is 66%, which is about 6.9% higher than single neural network.

The conventional procedure of skin-related disease detection is a visual inspection by a dermatologist or a primary care clinician, using a dermatoscope. The suspected patients with early signs of skin cancer are referred for biopsy and histopathological examination to ensure the correct diagnosis and the best treatment. Recent advancements in deep convolutional neural networks (CNNs) have achieved excellent performance in automated skin cancer classification with accuracy similar to that of dermatologists. However, such improvements are yet to bring about a clinically trusted and popular system for skin cancer detection. This study aimed to propose viable deep learning (DL) based method for the detection of skin cancer in lesion images, to help physicians in diagnosis. In this analytical study, a novel DL based model was proposed, in which other than the lesion image, the patient's data, including the anatomical site of the lesion, age, and gender were used as the model input to predict the type of the lesion. An Inception-ResNet-v2 CNN pretrained for object recognition was employed in the proposed model. Based on the results, the proposed method achieved promising performance for various skin conditions, and also using the patient's metadata in addition to the lesion image for classification improved the classification accuracy by at least 5% in all cases investigated. On a dataset of 57536 dermoscopic images, the proposed approach achieved an accuracy of 89.3%±1.1% in the discrimination of 4 major skin conditions and 94.5%±0.9% in the classification of benign vs. malignant lesions. The promising results highlight the efficacy of the proposed approach and indicate that the inclusion of the patient's metadata with the lesion image can enhance the skin cancer detection performance.

Achieving acoustically optimized structural design for the Guqin is crucial, combining its acoustic quality evaluation with craftsmanship. This paper proposes a parametric design method for non-uniform thickness distribution in the Guqin soundboard, enabling segmented thickness control and establishing 26 experimental models. Acoustic-structural coupled modal analysis models were constructed, and the acoustic quality of soundboards with varying thicknesses was evaluated. By comparing coupled modal characteristics and key acoustic parameters, the impact of panel structure on acoustic quality was quantitatively analyzed. The study reveals that adjusting the panel thickness distribution (±10 mm) alters the dominant vibration modes and their distribution, thereby influencing Guqin’s acoustic characteristics. To characterize the effect of all thickness distributions on acoustic quality and validate the correlations among thickness distribution, modal frequencies, and acoustic parameters, a multi-modal deep neural network (Mm-DNN) prediction model was developed. It integrates three subnetworks for predicting modal frequencies, acoustic parameters, and modal-acoustic parameters. The root mean square error of the model was < 0.13, and the loss value was < 0.52 × 10−2. Validation confirms Mm-DNN’s effectiveness in predicting Guqin’s acoustic performance from structural parameters. This research supports the industrialized manufacturing of Guqin and provides a digital design approach for predicting acoustic quality in traditional instruments.

One of the deadliest diseases is skin cancer, especially melanoma. The high resemblance between different skin lesions such as melanoma and nevus in the skin colour images increases the complexity of identification and diagnosis. An efficient automated early detection system for skin cancer detection is essential in order to save human lives, time, and effort. In this article, an automatic skin lesion classification system using a pretrained deep learning network and transfer learning was proposed. Here, diagnosing melanoma in premature stages, a detection system has been designed which contains the following digital image processing techniques. First, dermoscopy images of skin were taken and this is subjected to a preprocessing step for noise removal and postprocessing step for image enhancement. Then the processed image undergoes image segmentation using k-means and modified k-means clustering. Second, using feature extraction technology, Gray Level Co-occurrence Matrix, and first order statistics, characteristics are extracted. Features are selected on the basis of Harris Hawks optimization (HHO). Finally, various classifiers are used for predicting the stages and efficiency of the proposed work. Measures of well-known quantities, sensitivity, precision, accuracy, and specificity are used in assessing the efficiency of the suggested method, where higher values were obtained. Compared to the current methods, it is found that the classification rate exceeded the output of the current approaches in the performance of the proposed approach.

The objective of this study is to develop a multimodal neural network (MMNN) model that analyzes clinical variables and MRI images of a soft tissue sarcoma (STS) patient, to predict overall survival and risk of distant metastases. We compare the performance of this MMNN to models based on clinical variables alone, radiomics models, and an unimodal neural network. We include patients aged 18 or older with biopsy-proven STS who underwent primary resection between January 1st, 2005, and December 31st, 2020 with complete outcome data and a pre-treatment MRI with both a T1 post-contrast sequence and a T2 fat-sat sequence available. A total of 9380 MRI slices containing sarcomas from 287 patients are available. Our MMNN accepts the entire 3D sarcoma volume from T1 and T2 MRIs and clinical variables. Gradient blending allows the clinical and image sub-networks to optimally converge without overfitting. Heat maps were generated to visualize the salient image features. Our MMNN outperformed all other models in predicting overall survival and the risk of distant metastases. The C-Index of our MMNN for overall survival is 0.77 and the C-Index for risk of distant metastases is 0.70. The provided heat maps demonstrate areas of sarcomas deemed most salient for predictions. Our multimodal neural network with gradient blending improves predictions of overall survival and risk of distant metastases in patients with soft tissue sarcoma. Future work enabling accurate subtype-specific predictions will likely utilize similar end-to-end multimodal neural network architecture and require prospective curation of high-quality data, the inclusion of genomic data, and the involvement of multiple centers through federated learning.

Skin cancer segmentation is a critical task in a clinical decision support system for skin cancer detection. The suggested enhanced cuckoo search based optimization model will be used to evaluate several metrics in the skin cancer picture segmentation process. Because time and resources are always limited, the proposed enhanced cuckoo search optimization algorithm is one of the most effective strategies for dealing with global optimization difficulties. One of the most significant requirements is to design optimal solutions to optimize their use. There is no particular technique that can answer all optimization issues. The proposed enhanced cuckoo search optimization method indicates a constructive precision for skin cancer over with all image segmentation in computerized diagnosis. The accuracy of the proposed enhanced cuckoo search based optimization for melanoma has increased with a 23% to 29% improvement than other optimization algorithm. The total sensitivity and specificity attained in the proposed system are 99.56% and 99.73% respectively. The proposed method outperforms by offering accuracy of 99.26% in comparisons to other conventional methods. The proposed enhanced optimization technique achieved 98.75%, 98.96% for Dice and Jaccard coefficient. The model trained using the suggested measure outperforms those trained using the conventional method in the segmentation of skin cancer picture data.

A computer aided system for skin cancer detection based on Developed version of the Archimedes Optimization algorithm

Skin cancer can spread fast to nearby tissue and other parts of the human body if it's not diagnosed early. Most are curable only if skin cancer is found and treated in the early stages. Therefore, it's essential to seek a casual way of early diagnosis. This paper assesses a prototype system for skin cancer detection using an Arduino with an ArduCam Mega 5MP, benchmarked against smartphone diagnosis. Bandpass filters capture skin images at red (650 nm), green (532 nm), and blue (450 nm) wavelengths, measuring reflectance values. The approach aims to quantitatively determine skin melanin, oxyhemoglobin, and deoxyhemoglobin levels, aiding in various skin lesions' diagnosis. Evaluation involves comparing pixel reflectance values of images taken by smartphones and the prototype using a 3D mesh grid. Applying the modified Lambert-Beer law to reflectance values of moles, pimples, scars, scabs, and traces predicts relative levels of skin components. The system shows an 87% match with the smartphone standard, demonstrating high reliability. Further study might be needed to clarify the confirmation with clinical cases.