Non-image Information Research Articles

Detection support of lesions in patients with prostate cancer using -PSMA 1007 PET/CT.

This study proposes a detection support system for primary and metastatic lesions of prostate cancer using -PSMA 1007 positron emission tomography/computed tomography (PET/CT) images with non-image information, including patient metadata and location information of an input slice image. A convolutional neural network with condition generators and feature-wise linear modulation (FiLM) layers was employed to allow input of not only PET/CT images but also non-image information, namely, Gleason score, flag of pre- or post-prostatectomy, and normalized z-coordinate of an input slice. We explored the insertion position of the FiLM layers to optimize the conditioning of the network using non-image information. -PSMA 1007 PET/CT images were collected from 163 patients with prostate cancer and applied to the proposed system in a threefold cross-validation manner to evaluate the performance. The proposed system achieved a Dice score of 0.5732 (per case) and sensitivity of 0.8200 (per lesion), which are 3.87 and 4.16 points higher than the network without non-image information. This study demonstrated the effectiveness of the use of non-image information, including metadata of the patient and location information of the input slice image, in the detection of prostate cancer from -PSMA 1007 PET/CT images. Improvement in the sensitivity of inactive and small lesions remains a future challenge.

Label-Aware Dual Graph Neural Networks for Multi-Label Fundus Image Classification.

Fundus disease is a complex and universal disease involving a variety of pathologies. Its early diagnosis using fundus images can effectively prevent further diseases and provide targeted treatment plans for patients. Recent deep learning models for classification of this disease are gradually emerging as a critical research field, which is attracting widespread attention. However, in practice, most of the existing methods only focus on local visual cues of a single image, and ignore the underlying explicit interaction similarity between subjects and correlation information among pathologies in fundus diseases. In this paper, we propose a novel label-aware dual graph neural networks for multi-label fundus image classification that consists of population-based graph representation learning and pathology-based graph representation learning modules. Specifically, we first construct a population-based graph by integrating image features and non-image information to learn patient's representations by incorporating associations between subjects. Then, we represent pathologies as a sparse graph where its nodes are associated with pathology-based feature vectors and the edges correspond to probability of the co-occurrence of labels to generate a set of classifier scores by the propagation of multi-layer graph information. Finally, our model can adaptively recalibrate multi-label outputs. Detailed experiments and analysis of our results show the effectiveness of our method compared with state-of-the-art multi-label fundus image classification methods.

Photoreception and phototransduction in human skin

Light allows us to see. Seeing begins with photoreceptors in the retina where light is absorbed, converted into an electrical response and transmitted to the brain. Photoreception is highly conserved, and in animals, almost exclusively based on a single class of proteins, the opsins. Image-forming information goes to the lateral geniculate nucleus and eventual processing into visual images, sight. What may not be immediately apparent is the transmission of non-image information. However, cells in the skin contain the full set of opsins and their phototransduction cascades which are active, cover the whole solar spectrum and appear to participate in the skin’s protective mechanisms. This is obviously related to collection of non-image information. Why is this system of photoreception and transmission duplicated in the skin and what is its function?

MSA-GCN: A Multi-information Selection Aggregation Graph Convolutional Network for Breast Tumor Grading.

Physicians typically combine multi-modal data to make a graded diagnosis of breast tumors. However, most existing breast tumor grading methods rely solely on image information, resulting in limited accuracy in grading. This paper proposes a Multi-information Selection Aggregation Graph Convolutional Networks (MSA-GCN) for breast tumor grading. Firstly, to fully utilize phenotypic data reflecting the clinical and pathological characteristics of tumors, an automatic combination screening and weight encoder is proposed for phenotypic data, which can construct a population graph with improved structural information. Then, a graph structure is designed through similarity learning to reflect the correlation between patient image features. Finally, a multi-information selection aggregation mechanism is employed in the graph convolution model to extract the effective features of multi-modal data and enhance the classification performance of the model. The proposed method is evaluated on different clinical datasets from the Digital Database for Screening Mammography (DDSM) and INbreast. The average classification accuracies are 90.74% and 85.35%, respectively, surpassing the performance of existing methods. In conclusion, our method effectively fuses image and non-image information, leading to a significant improvement in the accuracy of breast tumor grading.

Fused Brain Functional Connectivity Network and Edge-attention Graph Convolution Network for Fibromyalgia Syndrome Diagnosis.

Fibromyalgia syndrome (FMS) is a type of rheumatology that seriously affects the normal life of patients. Due to the complex clinical manifestations of FMS, it is challenging to detect FMS. Therefore, an automatic FMS diagnosis model is urgently needed to assist physicians. Brain functional connectivity networks (BFCNs) constructed by resting-state functional magnetic resonance imaging (rs-fMRI) to describe brain functions have been widely used to identify individuals with relevant diseases from normal control (NC). Therefore, we propose a novel model based on BFCN and graph convolutional network (GCN) for automatic FMS diagnosis. Firstly, a novel fused BFCN method is proposed by fusing Pearson's correlation (PC) and low-rank (LR) BFCN, which retains information and reduces data redundancy to construct BFCN. Then we combine the feature of BFCN with non-image information of subjects to obtain nodes and adjacency matrices, which builds a graph with edge attention. Finally, the graph is sent to the GCN layer for FMS diagnosis. Our model is evaluated on the in-house FMS dataset to achieve 82.48% accuracy. The experimental results show that our method outperforms the state-of-the-art competing methods.

An extensible hierarchical graph convolutional network for early Alzheimer’s disease identification

Background and Objective: For early identification of Alzheimer’s disease (AD) based on multi-modal magnetic resonance imaging (MRI) data, it is important to make comprehensive use of image features and non-image information to analyze the gray matter atrophy and the structural/functional connectivity abnormalities for different courses of AD.Methods: In this study, we propose an extensible hierarchical graph convolutional network (EH-GCN) for early AD identification. Based on the extracted image features from multi-modal MRI data using the presented multi-branch residual network (ResNet), the brain regions-of-interests (ROIs) based GCN is built to extract structural and functional connectivity features between different ROIs of the brain. In order to further improve the performance of AD identification, an optimized spatial GCN is proposed as convolution operator in the population-based GCN to avoid rebuilding the graph network and take advantage of relationships between subjects. Finally, the proposed EH-GCN is built by embedding the image features and internal brain connectivity features into the spatial population-based GCN, which provides an extensible way to improve early AD identification performance by adding imaging features and non-image information from multi-modal data.Results: Experiments are performed on two datasets, which illustrate the effectiveness of the extracted structural/functional connectivity features and the high computational efficiency of the proposed method. The classification accuracy of AD vs NC, AD vs MCI and MCI vs NC classification tasks reaches 88.71%, 82.71% and 79.68% respectively. The extracted connectivity features between ROIs indicate that functional abnormalities are earlier than gray matter atrophy and abnormalities of structural connections, which is consistent with the clinical manifestations. The proposed method allows for the addition of other modal image features and non-image information from multi-modal data to continuously improve the performance of clinical data analysis.Conclusions: The proposed method can help us comprehensively analyze the role of gray matter atrophy, the damage of white matter nerve fiber tracts and the degradation of functional connectivity for different courses of AD, which could be useful for further extraction of clinical biomarkers for early AD identification.

Counterfactual inference graph network for disease prediction

Graph convolutional networks are widely used as computational models that integrate the data of image and non-image modalities in the medical diagnostic domain, especially while exploring population-based disease prediction. However, existing methods use associative combinations of non-image information while constructing population graphs, which are often interfered while capturing the relation between subjects, and the causal relationship between these variables is ignored. In this study, we present a counterfactual inference graph network for improving disease prediction accuracy and generalization by constructing a learnable population relational graph structure. Specifically, nodes can be regarded as contexts, and edge (non-)existence and node classification can be regarded as treatments and outcomes, respectively. We enhance graph learning by predicting both, the observed factual classifications and counterfactual classifications. To provide an estimate of the prediction uncertainty associated with relational graphs, a novel concept called edge uncertainty was proposed. Through extensive experiments on six publicly available datasets on four types of diseases: autism spectrum disorder (ASD), Alzheimer’s disease (AD), thoracic diseases, and ocular diseases, we demonstrate that our proposed method can significantly and consistently improve the prediction performance compared with baseline approaches.

Multi-channel content based image retrieval method for skin diseases using similarity network fusion and deep community analysis

Content-based image retrieval (CBIR) could be an efficient diagnostic tool. Physicians could consult a CBIR system before making a diagnosis for a clinical case by retrieving a set of images with similar appearance and pathological diagnosis from a data archive. With access to various imaging modalities, physicians may want to match more than one image modality and non-image information. How to make full use of this diverse information is an important research question. In this paper, we propose a CBIR framework for skin diseases that incorporates multi-sourced information including dermoscopic images, clinical images, and meta information. The proposed framework fuses the multi-sourced features in mutual similarity level; thus, solving severe dimensional bias problems for image and non-image information. We then utilize a graph-based community analysis on similarity networks where similar images are strongly connected and help retrieve similar images with improved performance. Evaluations were carried out using two well-known skin datasets EDRA and ISIC 2019. The carefully designed framework demonstrates a substantial improvement in finding similar cases for different skin diseases with an average precision of 0.836, which is the state-of-the-art performance for retrieving skin disease types. In addition, the proposed framework is also applicable to scenarios with a single typed feature with improved performance. By integrating multi-sourced information from the same patient, the proposed CBIR system could be potentially used in complex clinical scenarios with a trustable performance benefitting from both abundant information and advanced community search technique.

A high-generalizability machine learning framework for predicting the progression of Alzheimer\u2019s disease using limited data

Alzheimer’s disease is a neurodegenerative disease that imposes a substantial financial burden on society. A number of machine learning studies have been conducted to predict the speed of its progression, which varies widely among different individuals, for recruiting fast progressors in future clinical trials. However, because the data in this field are very limited, two problems have yet to be solved: the first is that models built on limited data tend to induce overfitting and have low generalizability, and the second is that no cross-cohort evaluations have been done. Here, to suppress the overfitting caused by limited data, we propose a hybrid machine learning framework consisting of multiple convolutional neural networks that automatically extract image features from the point of view of brain segments, which are relevant to cognitive decline according to clinical findings, and a linear support vector classifier that uses extracted image features together with non-image information to make robust final predictions. The experimental results indicate that our model achieves superior performance (accuracy: 0.88, area under the curve [AUC]: 0.95) compared with other state-of-the-art methods. Moreover, our framework demonstrates high generalizability as a result of evaluations using a completely different cohort dataset (accuracy: 0.84, AUC: 0.91) collected from a different population than that used for training.

Multi-dimension tool wear state assessment criterion on the spiral edge of the milling cutter

In the assessment of the tool wear state for the spiral edge of milling cutter based on machine vision, the traditional assessment criterion is often inaccurate due to the problem of missing of information, especially near the tip area. In order to deal with this problem, different lighting-condition settings and additional non-image information compensation techniques are needed. In view of this, an integration tool wear detection method combined line laser edge detection and machine vision is proposed. Subsequently, the combined data acquirement experimental system is designed and built, which can simultaneously acquire two types of parameters: diameter and images under the same detection condition. Then, based on the above information, a multi-dimension series assessment criterion is proposed consisting of three dimensions index: The one-dimensional assessment index gives the average wear value of the spiral side, which is mainly used as data ordination and characterizes the rule of tool wear degradation; the two-dimensional assessment index describes the contour of the wear region and calculates the area value, which can predict the change of the tool wear stage more precisely; and the three-dimensional assessment index gives the 3D morphology by adding depth information within the wear region and quantifies the volume of the worn-off part, which provides clues for early warning of the deterioration caused by the tool worn. Experiments proved that the multi-dimension series assessment criterion is helpful in reflecting the trend of tool life and giving a more accurate assessment of the tool wear state for the spiral edge of a milling cutter.

Deep Learning for Automated Diabetic Retinopathy Screening Fused With Heterogeneous Data From EHRs Can Lead to Earlier Referral Decisions.

PurposeFundus images are typically used as the sole training input for automated diabetic retinopathy (DR) classification. In this study, we considered several well-known DR risk factors and attempted to improve the accuracy of DR screening.MetphodsFusing nonimage data (e.g., age, gender, smoking status, International Classification of Disease code, and laboratory tests) with data from fundus images can enable an end-to-end deep learning architecture for DR screening. We propose a neural network that simultaneously trains heterogeneous data and increases the performance of DR classification in terms of sensitivity and specificity. In the current retrospective study, 13,410 fundus images and their corresponding nonimage data were collected from the Chung Shan Medical University Hospital in Taiwan. The images were classified as either nonreferable or referable for DR by a panel of ophthalmologists. Cross-validation was used for the training models and to evaluate the classification performance.ResultsThe proposed fusion model achieved 97.96% area under the curve with 96.84% sensitivity and 89.44% specificity for determining referable DR from multimodal data, and significantly outperformed the models that used image or nonimage information separately.ConclusionsThe fusion model with heterogeneous data has the potential to improve referable DR screening performance for earlier referral decisions.Translational RelevanceArtificial intelligence fused with heterogeneous data from electronic health records could provide earlier referral decisions from DR screening.

Classification of Mild Cognitive Impairment Based on a Combined High-Order Network and Graph Convolutional Network

Detection of early stages of Alzheimer's disease (AD) (i.e., mild cognitive impairment (MCI)) is important because it can delay or prevent progression to AD. The current researches of MCI classification are mainly based on static low-order functional connectivity network (FCN) and image information. However, static FCN cannot reflect time-varying dynamic behavior, low-order FCN overlooks inter-region interactions, and ignoring non-image information is not suitable especially when the size of dataset is small. In this paper, a method based on a combined high-order network and graph convolutional network (GCN) is proposed. The combined high-order network combines static, dynamic and high-level information to construct FCN while GCN is used to include non-image information to improve classifier's performance. Firstly, dynamic FCNs and static FCN are constructed by using a sliding window approach. Secondly, dynamic high-order FCNs and static high-order FCN based on the topographical similarity are then constructed. Thirdly, a novel combination method is proposed to utilize dynamic high-order FCNs and static high-order FCN to form a combined high-order FCN. Fourthly, features of the combined high-order FCNs are extracted by using a recursive feature elimination method. Lastly, after inputting extracted features into the GCN, in which MCI-graph establishes interactions between individuals and populations by using non-image information, the GCN outputs the binary classification results. Experimental results on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset (adni.loni.ucla.edu) show that our framework has good performance.

Automatic Flare Spot Artifact Detection and Removal in Photographs

Flare spot is one type of flare artifact caused by a number of conditions, frequently provoked by one or more high-luminance sources within or close to the camera field of view. When light rays coming from a high-luminance source reach the front element of a camera, it can produce intra-reflections within camera elements that emerge at the film plane forming non-image information or flare on the captured image. Even though preventive mechanisms are used, artifacts can appear. In this paper, we propose a robust computational method to automatically detect and remove flare spot artifacts. Our contribution is threefold: firstly, we propose a characterization which is based on intrinsic properties that a flare spot is likely to satisfy; secondly, we define a new confidence measure able to select flare spots among the candidates; and, finally, a method to accurately determine the flare region is given. Then, the detected artifacts are removed by using exemplar-based inpainting. We show that our algorithm achieves top-tier quantitative and qualitative performance.

I095] Patient dosimetry in interventional radiology: concepts and tools

Biological effects induced by exposure to radiation made evident the need for an accurate dosimetry from the early days of X-ray use in medicine. Later the scientific interest turned also to the estimation of risks of radiation-induced cancer and genetic effects. Various dosimetric quantities have been used in diagnostic radiology and sometimes confusion is caused because the same name is used for different quantities. The question arises: what is patient dose and what is not? It’s common practice to indicate the dose received by a patient in radiology using the dosimetric indexes defined for the different imaging modalities. However, dosimetric indexes are a useful tool for practices optimization and comparison, but certainly they don’t describe patient dose. To accurately estimate patient dose there is the need to collect a lot of information concerning patient characteristics and irradiation events in order to obtain organ or tissue dose. Nowadays, thanks to the digital technology used in medical imaging, various ways of collecting these data are available. The information is communicated using specific standards (i.e. DICOM), such as the non-image information object definitions. These objects are carrying information about equipment output and other dose related information which allow the user to provide a more accurate estimate of patient dose. The standards are in continuous development, to provide more and more detailed information and to include most recent modalities (e.g. CBCT). This talk will review the basic concepts on patient dosimetry as well as the existing DICOM objects and IHE profiles used to monitor patient exposure, highlighting pros and cons and introducing to the future developments.

DICOM standards for patient dosimetry

Data related to patient radiation exposure during radiologic procedures can be stored and communicated using the non-image information object definitions (non IOD) DICOM objects. More specifically, there are three DICOM objects in the standard that can be used to transport exposure information: the DICOM header, the DICOM Modality Performed Procedure Step (MPPS) and the DICOM Radiation Dose Structured Report (RDSR). All these objects have several advantages and some limitations. All the mentioned objects are carrying information about equipment output and do not provide any information on patient dose. The most complete of the three objects previously mentioned is the DICOM RDSR, which was added to the Standard in 2005. This object transports the information related to the equipment output along with information about the patient and the procedure that could be used to estimate the exposure to the patient. Although unfortunately this information is not complete. Moreover some new modalities (e.g: CBCT) need specific attributes to be stored and communicated and the present RDSR structure needs to be enhanced to allow it. Furthermore none of this DICOM object includes a template for patient-specific dose data to be recorded after estimation. For this reason a new DICOM object, the Patient Radiation Dose Structured Report (P-RDSR) has being developed. The new P-RDSR will provide the archival tool to record the results of the estimations and the methodology used to achieve this estimate.

The DICOM Radiation Dose Structured Report: What It Is and What It Is Not

Although most radiologists and medical physicists have heard of the term “DICOM,” which stands for Digital Imaging and Communications inMedicine,many associate this term onlywith the format used to save radiographic images. However, the DICOM standard actually includes the specifications for communicating many types of medical-related data in an organized manner so that devices from different hardware and software manufacturers can talk to one another [1]. Examples of these different types of data, called nonimage information object definitions (IODs) in DICOMspeak, are treatment plans, waveforms, and structured reports, among others. Over the past few years, there has been an increasing awareness of the importance of the radiation dose used to acquire images with ionizing radiation, and with it there has been an increase in the efforts to optimize the levels of radiation used. For this, it is important to be able to store and analyze data regarding the exposure involved during these acquisitions in a standard format that can be machine read and interpreted. Although exposure information is recorded in the header of each image, this solution is not complete or ideal for various reasons. In the first place, there is no standardmethod to decouple the image itself from the exposure information. Therefore, to access and store the exposure information, it must be accompanied by the image data,

High-throughput neuro-imaging informatics

This paper describes neuroinformatics technologies at 1 mm anatomical scale based on high-throughput 3D functional and structural imaging technologies of the human brain. The core is an abstract pipeline for converting functional and structural imagery into their high-dimensional neuroinformatic representation index containing O(1000–10,000) discriminating dimensions. The pipeline is based on advanced image analysis coupled to digital knowledge representations in the form of dense atlases of the human brain at gross anatomical scale. We demonstrate the integration of these high-dimensional representations with machine learning methods, which have become the mainstay of other fields of science including genomics as well as social networks. Such high-throughput facilities have the potential to alter the way medical images are stored and utilized in radiological workflows. The neuroinformatics pipeline is used to examine cross-sectional and personalized analyses of neuropsychiatric illnesses in clinical applications as well as longitudinal studies. We demonstrate the use of high-throughput machine learning methods for supporting (i) cross-sectional image analysis to evaluate the health status of individual subjects with respect to the population data, (ii) integration of image and personal medical record non-image information for diagnosis and prognosis.

Matching clinical and biological needs with emerging imaging technologies

Acquisition, analysis, and integration of medical images with nonimage biomedical information are essential for research and clinical practice. Individualization of health care, assessment of disease risk, tailoring of therapies, and lifelong monitoring places extraordinary demands on emerging technologies in medical imaging, information management, and molecular medicine. To achieve the anticipated benefits in outcomes that new technology can deliver, imaging informatics infrastructure and tools must be available to represent, store, integrate, and present the data in a useful manner. Currently, images are not well integrated with other forms of biological knowledge. This paper outlines the clinical and research requirements for imaging, and identifies barriers to progress and links these needs with the current and future informatics technologies that may overcome them.

Computer-assisted diagnosis of breast cancer using a data-driven Bayesian belief network

This study investigates a simple Bayesian belief network for the diagnosis of breast cancer, and specifically addresses the question of whether integrating image and non-image based features into a single network can yield better performance than hybrid combinations of independent networks. From a dataset of 419 cases, including 92 malignancies, 13 features relating to mammographic findings, physical examinations and patients' clinical histories, were extracted to build three Bayesian belief networks. The scenarios tested included a network incorporating all features and two hybrids which combined the outputs of sub-networks corresponding to the image or non-image features. Average areas ( A z ) under the corresponding ROC curves were used as measures of performance. The network incorporating only image based features performed better ( A z =0.81) than that using nonimage features ( A z =0.71). Both hybrid classifiers yielded better performance ( A z =0.85 for averaging and A z =0.87 for logistic regression), but neither hybrid was as accurate as the network incorporating all features ( A z =0.89). This preliminary study suggests that, like human observers who concurrently consider different types of information, a single classifier that simultaneously evaluates both image and non-image information can achieve better diagnostic performance than the hybrid combinations considered here.