Large Annotated Datasets Research Articles

Efficient Cancer Classification by Coupling Semi Supervised and Multiple Instance Learning

The annotation of large datasets is often the bottleneck in the successful application of artificial intelligence in computational pathology. For this reason recently Multiple Instance Learning (MIL) and Semi Supervised Learning (SSL) approaches are gaining popularity because they require fewer annotations. In this work we couple SSL and MIL to train a deep learning classifier that combines the advantages of both methods and overcomes their limitations. Our method is able to learn from the global WSI diagnosis and a combination of labeled and unlabeled patches. Furthermore, we propose and evaluate an efficient labeling paradigm that guarantees a strong classification performance when combined with our learning framework. We compare our method to SSL and MIL baselines, the state-of-the-art and completely supervised training. With only a small percentage of patch labels our proposed model achieves a competitive performance on SICAPv2 (Cohen’s kappa of 0.801 with 450 patch labels), PANDA (Cohen’s kappa of 0.794 with 22,023 patch labels) and Camelyon16 (ROC AUC of 0.913 with 433 patch labels). Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/arneschmidt/ssl_and_mil_cancer_classification</uri> .

AUTOMATIC EFFUSION MEASUREMENT WITH LIMITED LABELED DATA

Accurate quantification of effusion volume is becoming more important as OA is increasingly recognized to have inflammatory components including joint effusion. Effective clinical management requires accurate quantification of features related to inflammation; particularly synovitis and joint effusion which represents a target for therapy. Rather than relying on highly variable subjective assessment of joint effusions, with the development of deep learning techniques we have witnessed increased automation of radiological assessments. However, most of these techniques use supervised learning and are trained on large annotated datasets. Acquiring “ground truth” labels for large datasets is expensive and time-consuming. This becomes even more complicated and challenging when it comes to detecting medical image features like effusion. Detecting effusions accurately requires a high level of expertise. Because pockets of joint fluid can be in unexpected locations, human experts frequently need to step back to look at the entire image stack multiple times to prevent inattentional error. Therefore, proposing a method for learning from limited labeled data is crucial for medical image analysis. To perform automated measurement of knee effusion volume using minimal labeled data. In this method, we aim to develop an alternative deep learning training strategy based on self-supervised pretraining on unlabeled knee MRI scans. This study requires two sets of data: 1) a large unlabeled dataset, and 2) a small labeled dataset. For the large unlabeled dataset, we have used 4 different OAI MRI sequence data (64k slices have been selected randomly). For the labeled dataset we have used low-resolution sagittal Turbo Spin Echo (TSE) MR sequences from the OAI dataset. Effusion regions were extracted by a trained musculoskeletal radiologist (DC) for 23 participants (a total of 31 scans) using an interactive software developed in-house. We have used 5 scans for the test and 26 scans (a total of 700 slices) for training. In this method, IMaskRCNN deep learning architecture with the ResNet-101 backbone have been used. The training process has been performed in two-phase, self-supervised pretraining, and fine-tuning. The self-supervised pretraining using the fill-blank method was performed on unlabeled images to learn visual representations in musculoskeletal MRIs. After pretraining, we fined-tuned last layers of the network (backbone weights have been frozen) using the training division of the small TSE dataset. The dice similarity coefficient, Intersection over union (IoU) was used to compare the result of the segmentation between the predicted output and the ground truth. Qualitative results of the proposed method is presented in the figure. As shown the network was able to provide a good prediction of effusion location and extracting it. The segmentation model using the proposed pretraining method was successful in finding the location of effusion for 90% of the slices with IoU = 60%. The dice score for the detected effusion is 0.70 indicates a high agreement between the automatic segmentation and the labels provided by the expert. Self-supervised pretraining on the OAI dataset, can be a solution to the scarcity of pixel-wise ground truth (GT) for effusion quantification. Alberta Innovates, AHS Chair in Diagnostic Imaging, Medical Imaging Consultants, CIHR. CORRESPONDENCE ADDRESS: banafshe.felfeliyan@ucalgary.ca

Semi-Supervised Skin Lesion Segmentation With Coupling CNN and Transformer Features

An automatic skin lesion segmentation algorithm not only facilitates the dermatologist’s workload on skin cancer analysis but also provides a platform for early cancer prediction. Over the years, several deep learning methods have been proposed to address the skin lesion segmentation problem. However, training deep models usually requires a large-scale annotated dataset, which is not feasible in the medical domain due to the annotation burden. In addition, the low data regime highly increases the overfitting potential for the neural network. To address these limitations in an end-to-end manner, we propose to incorporate unlabelled samples during the training process. Our network offers a semi-supervised training schema, wherein the first stage performs a supervised training strategy to learn semantic segmentation map while the second step focuses on the unsupervised technique to enrich the encoder module. Specifically, unlike the literature work on skin lesion segmentation, we design a surrogate task on top of the convolutional and Transformer representations to learn data-driven features from the image itself to alleviate the requirement of the large annotated dataset. The effectiveness of the proposed method is demonstrated using three different skin lesion segmentation datasets, namely ISIC 2018 (dice score 0.905), ISIC 2017 (dice score 0.898) and PH2 (dice score 0.940). Particularly we observed that including the unsupervised samples can increase the dice score by 2%.

Graphical integrity issues in open access publications: Detection and patterns of proportional ink violations.

Academic graphs are essential for communicating complex scientific ideas and results. To ensure that these graphs truthfully reflect underlying data and relationships, visualization researchers have proposed several principles to guide the graph creation process. However, the extent of violations of these principles in academic publications is unknown. In this work, we develop a deep learning-based method to accurately measure violations of the proportional ink principle (AUC = 0.917), which states that the size of shaded areas in graphs should be consistent with their corresponding quantities. We apply our method to analyze a large sample of bar charts contained in 300K figures from open access publications. Our results estimate that 5% of bar charts contain proportional ink violations. Further analysis reveals that these graphical integrity issues are significantly more prevalent in some research fields, such as psychology and computer science, and some regions of the globe. Additionally, we find no temporal and seniority trends in violations. Finally, apart from openly releasing our large annotated dataset and method, we discuss how computational research integrity could be part of peer-review and the publication processes.

Digital Forensics Use Case for Glaucoma Detection Using Transfer Learning Based on Deep Convolutional Neural Networks

In parallel with the development of various emerging fields such as computer vision and related technologies, e.g., iris identification and glaucoma detection, criminals are developing their methods. It is the foremost reason for the blindness of human beings that affects the eye’s optic nerve. Fundus photography is carried out to examine this eye disease. Medical experts evaluate fundus photographs, which is a time-consuming visual inspection. Most current systems for automated glaucoma detection in fundus images rely on segmentation-based features nuanced by the underlying segmentation methods. Convolutional neural networks (CNNs) are powerful tools for solving image classification tasks, as they can learn highly discriminative features from raw pixel intensities. However, their applicability to medical image analysis is limited by the nonavailability of large sets of annotated data required for training. In this work, we aim to accelerate this process using a computer-aided diagnosis of this severe disease with the help of transfer learning based on deep convolutional neural networks. We have suggested the Inception V-3 approach for image classification based on convolution neural networks. Our developed model has the potential to address this CNN model’s problem of classification accuracy, and with imaging data, our proposed method outperforms recent state-of-the-art approaches. The case study for digital forensics is an essential component of emerging technologies, and hence glaucoma detection plays a vital role in it.

CNN Filter Learning from Drawn Markers for the Detection of Suggestive Signs of COVID-19 in CT Images.

Early detection of COVID-19 is vital to control its spread. Deep learning methods have been presented to detect suggestive signs of COVID-19 from chest CT images. However, due to the novelty of the disease, annotated volumetric data are scarce. Here we propose a method that does not require either large annotated datasets or backpropagation to estimate the filters of a convolutional neural network (CNN). For a few CT images, the user draws markers at representative normal and abnormal regions. The method generates a feature extractor composed of a sequence of convolutional layers, whose kernels are specialized in enhancing regions similar to the marked ones, and the decision layer of our CNN is a support vector machine. As we have no control over the CT image acquisition, we also propose an intensity standardization approach. Our method can achieve mean accuracy and kappa values of 0.97 and 0.93, respectively, on a dataset with 117 CT images extracted from different sites, surpassing its counterpart in all scenarios.

Segmentation of Aorta 3D CT Images Based on 2D Convolutional Neural Networks

The automatic segmentation of the aorta can be extremely useful in clinical practice, allowing the diagnosis of numerous pathologies to be sped up, such as aneurysms and dissections, and allowing rapid reconstructive surgery, essential in saving patients’ lives. In recent years, the success of Deep Learning (DL)-based decision support systems has increased their popularity in the medical field. However, their effective application is often limited by the scarcity of training data. In fact, collecting large annotated datasets is usually difficult and expensive, especially in the biomedical domain. In this paper, an automatic method for aortic segmentation, based on 2D convolutional neural networks (CNNs), using 3D CT (computed axial tomography) scans as input is presented. For this purpose, a set of 153 CT images was collected and a semi-automated approach was used to obtain their 3D annotations at the voxel level. Although less accurate, the use of a semi-supervised labeling technique instead of a full supervision proved necessary to obtain enough data in a reasonable amount of time. The 3D volume was analyzed using three 2D segmentation networks, one for each of the three CT views (axial, coronal and sagittal). Two different network architectures, U-Net and LinkNet, were used and compared. The main advantages of the proposed method lie in its ability to work with a reduced number of data even with noisy targets. In addition, analyzing 3D scans based on 2D slices allows for them to be processed even with limited computing power. The results obtained are promising and show that the neural networks employed can provide accurate segmentation of the aorta.

Landmark Dataset Development and Recognition

Landmark recognition aims to detect popular natural and manmade structures within an image. It is challenging with one of the reasons being the lack of large annotated datasets. Existing work mainly focuses on landmarks located in Europe and North America due to regional and language bias. In this study, the authors build a comprehensive Chinese landmark dataset to complement the current data and to benefit research for landmark recognition. It is done by leveraging the vast amount of multimedia data on the web and utilizing image clustering and retrieval techniques in data preparation and analysis. This results in a Chinese landmark dataset with a total of 42,548 images for 987 unique landmarks. In addition, a landmark recognition model is developed based on advanced deep learning techniques and integrated into a mobile application that allows users to do landmark prediction without the need of internet access or cellular data coverage.

Visual Summary Identification From Scientific Publications via Self-Supervised Learning.

The exponential growth of scientific literature yields the need to support users to both effectively and efficiently analyze and understand the some body of research work. This exploratory process can be facilitated by providing graphical abstracts–a visual summary of a scientific publication. Accordingly, previous work recently presented an initial study on automatic identification of a central figure in a scientific publication, to be used as the publication’s visual summary. This study, however, have been limited only to a single (biomedical) domain. This is primarily because the current state-of-the-art relies on supervised machine learning, typically relying on the existence of large amounts of labeled data: the only existing annotated data set until now covered only the biomedical publications. In this work, we build a novel benchmark data set for visual summary identification from scientific publications, which consists of papers presented at conferences from several areas of computer science. We couple this contribution with a new self-supervised learning approach to learn a heuristic matching of in-text references to figures with figure captions. Our self-supervised pre-training, executed on a large unlabeled collection of publications, attenuates the need for large annotated data sets for visual summary identification and facilitates domain transfer for this task. We evaluate our self-supervised pretraining for visual summary identification on both the existing biomedical and our newly presented computer science data set. The experimental results suggest that the proposed method is able to outperform the previous state-of-the-art without any task-specific annotations.

Highly accurate differentiation of bone marrow cell morphologies using deep neural networks on a large image data set

AbstractBiomedical applications of deep learning algorithms rely on large expert annotated data sets. The classification of bone marrow (BM) cell cytomorphology, an important cornerstone of hematological diagnosis, is still done manually thousands of times every day because of a lack of data sets and trained models. We applied convolutional neural networks (CNNs) to a large data set of 171 374 microscopic cytological images taken from BM smears from 945 patients diagnosed with a variety of hematological diseases. The data set is the largest expert-annotated pool of BM cytology images available in the literature. It allows us to train high-quality classifiers of leukocyte cytomorphology that identify a wide range of diagnostically relevant cell species with high precision and recall. Our CNNs outcompete previous feature-based approaches and provide a proof-of-concept for the classification problem of single BM cells. This study is a step toward automated evaluation of BM cell morphology using state-of-the-art image-classification algorithms. The underlying data set represents an educational resource, as well as a reference for future artificial intelligence–based approaches to BM cytomorphology.

Anno-Mate: Human–Machine Collaboration Features for Fast Annotation

Large annotated datasets are crucial for training deep machine learning models, but they are expensive and time-consuming to create. There are already numerous public datasets, but a vast amount of unlabeled data, especially video data, can still be annotated and leveraged to further improve the performance and accuracy of machine learning models. Therefore, it is essential to reduce the time and effort required to annotate a dataset to prevent bottlenecks in the development of this field. In this study, we propose Anno-Mate, a pair of features integrated into the Computer Vision Annotation Tool (CVAT). It facilitates human–machine collaboration and reduces the required human effort. Anno-Mate comprises Auto-Fit, which uses an EfficientDet-D0 backbone to tighten an existing bounding box around an object, and AutoTrack, which uses a channel and spatial reliability tracking (CSRT) tracker to draw a bounding box on the target object as it moves through the video frames. Both features exhibit a good speed and accuracy trade-off. Auto-Fit garnered an overall accuracy of 87% and an average processing time of 0.47 s, whereas the AutoTrack feature exhibited an overall accuracy of 74.29% and could process 18.54 frames per second. When combined, these features are proven to reduce the time required to annotate a minute of video by 26.56%.

A Transfer Learning Architecture Based on a Support Vector Machine for Histopathology Image Classification

Recently, digital pathology is an essential application for clinical practice and medical research. Due to the lack of large annotated datasets, the deep transfer learning technique is often used to classify histopathology images. A softmax classifier is often used to perform classification tasks. Besides, a Support Vector Machine (SVM) classifier is also popularly employed, especially for binary classification problems. Accurately determining the category of the histopathology images is vital for the diagnosis of diseases. In this paper, the conventional softmax classifier and the SVM classifier-based transfer learning approach are evaluated to classify histopathology cancer images in a binary breast cancer dataset and a multiclass lung and colon cancer dataset. In order to achieve better classification accuracy, a methodology that attaches SVM classifier to the fully-connected (FC) layer of the softmax-based transfer learning model is proposed. The proposed architecture involves a first step training the newly added FC layer on the target dataset using the softmax-based model and a second step training the SVM classifier with the newly trained FC layer. Cross-validation is used to ensure no bias for the evaluation of the performance of the models. Experimental results reveal that the conventional SVM classifier-based model is the least accurate on either binary or multiclass cancer datasets. The conventional softmax-based model shows moderate classification accuracy, while the proposed synthetic architecture achieves the best classification accuracy.

A self-supervised learning strategy for postoperative brain cavity segmentation simulating resections

PurposeAccurate segmentation of brain resection cavities (RCs) aids in postoperative analysis and determining follow-up treatment. Convolutional neural networks (CNNs) are the state-of-the-art image segmentation technique, but require large annotated datasets for training. Annotation of 3D medical images is time-consuming, requires highly trained raters and may suffer from high inter-rater variability. Self-supervised learning strategies can leverage unlabeled data for training.MethodsWe developed an algorithm to simulate resections from preoperative magnetic resonance images (MRIs). We performed self-supervised training of a 3D CNN for RC segmentation using our simulation method. We curated EPISURG, a dataset comprising 430 postoperative and 268 preoperative MRIs from 430 refractory epilepsy patients who underwent resective neurosurgery. We fine-tuned our model on three small annotated datasets from different institutions and on the annotated images in EPISURG, comprising 20, 33, 19 and 133 subjects.ResultsThe model trained on data with simulated resections obtained median (interquartile range) Dice score coefficients (DSCs) of 81.7 (16.4), 82.4 (36.4), 74.9 (24.2) and 80.5 (18.7) for each of the four datasets. After fine-tuning, DSCs were 89.2 (13.3), 84.1 (19.8), 80.2 (20.1) and 85.2 (10.8). For comparison, inter-rater agreement between human annotators from our previous study was 84.0 (9.9).ConclusionWe present a self-supervised learning strategy for 3D CNNs using simulated RCs to accurately segment real RCs on postoperative MRI. Our method generalizes well to data from different institutions, pathologies and modalities. Source code, segmentation models and the EPISURG dataset are available at https://github.com/fepegar/resseg-ijcars.

Ultrasonic assessment of osseointegration phenomena at the bone-implant interface using convolutional neural network.

Although endosseous implants are widely used in the clinic, failures still occur and their clinical performance depends on the quality of osseointegration phenomena at the bone-implant interface (BII), which are given by bone ingrowth around the BII. The difficulties in ensuring clinical reliability come from the complex nature of this interphase related to the implant surface roughness and the presence of a soft tissue layer (non-mineralized bone tissue) at the BII. The aim of the present study is to develop a method to assess the soft tissue thickness at the BII based on the analysis of its ultrasonic response using a simulation based-convolution neural network (CNN). A large-annotated dataset was constructed using a two-dimensional finite element model in the frequency domain considering a sinusoidal description of the BII. The proposed network was trained by the synthesized ultrasound responses and was validated by a separate dataset from the training process. The linear correlation between actual and estimated soft tissue thickness shows excellent R2 values equal to 99.52% and 99.65% and a narrow limit of agreement corresponding to [ -2.56, 4.32 μm] and [ -15.75, 30.35 μm] of microscopic and macroscopic roughness, respectively, supporting the reliability of the proposed assessment of osseointegration phenomena.

Zero-Shot Action Recognition with Three-Stream Graph Convolutional Networks.

Large datasets are often used to improve the accuracy of action recognition. However, very large datasets are problematic as, for example, the annotation of large datasets is labor-intensive. This has encouraged research in zero-shot action recognition (ZSAR). Presently, most ZSAR methods recognize actions according to each video frame. These methods are affected by light, camera angle, and background, and most methods are unable to process time series data. The accuracy of the model is reduced owing to these reasons. In this paper, in order to solve these problems, we propose a three-stream graph convolutional network that processes both types of data. Our model has two parts. One part can process RGB data, which contains extensive useful information. The other part can process skeleton data, which is not affected by light and background. By combining these two outputs with a weighted sum, our model predicts the final results for ZSAR. Experiments conducted on three datasets demonstrate that our model has greater accuracy than a baseline model. Moreover, we also prove that our model can learn from human experience, which can make the model more accurate.

A deep learning approach for the automatic recognition of prosthetic mitral valve in echocardiographic images

The first step in the automatic evaluation of the cardiac prosthetic valve is the recognition of such valves in echocardiographic images. This research surveyed whether a deep convolutional neural network (DCNN) could improve the recognition of prosthetic mitral valve in conventional 2D echocardiographic images. An efficient intervention to decrease the misreading rate of the prosthetic mitral valve is required for non-expert cardiologists. This intervention could serve as a section of a fully-automated analysis chain, alleviate the cardiologist's workload, and improve precision and time management, especially in an emergent situation. Additionally, it might be suitable for pre-labeling large databases of unclassified images. We, therefore, introduce a large publicly-available annotated dataset for the purpose of prosthetic mitral valve recognition. We utilized 2044 comprehensive non-stress transthoracic echocardiographic studies. Totally, 1597 patients had natural mitral valves and 447 patients had prosthetic valves. Each case contained 1 cycle of echocardiographic images from the apical 4-chamber (A4C) and the parasternal long-axis (PLA) views. Thirteen versions of the state-of-the-art models were independently trained, and the ensemble predictions were performed using those versions. For the recognition of prosthetic mitral valves from natural mitral valves, the area under the receiver-operating characteristic curve (AUC) made by the deep learning algorithm was similar to that made by cardiologists (0.99). In this research, EfficientNetB3 architecture in the A4C view and the EfficientNetB4 architecture in the PLA view were the best models among the other pre-trained DCNN models.

A Transformer-based Framework for Neutralizing and Reversing the Political Polarity of News Articles

People often prefer to consume news with similar political predispositions and access like-minded news articles, which aggravates polarized clusters known as "echo chamber". To mitigate this phenomenon, we propose a computer-aided solution to help combat extreme political polarization. Specifically, we present a framework for reversing or neutralizing the political polarity of news headlines and articles. The framework leverages the attention mechanism of a Transformer-based language model to first identify polar sentences, and then either flip the polarity to the neutral or to the opposite through a GAN network. Tested on the same benchmark dataset, our framework achieves a 3%-10% improvement on the flipping/neutralizing success rate of headlines compared with the current state-of-the-art model. Adding to prior literature, our framework not only flips the polarity of headlines but also extends the task of polarity flipping to full-length articles. Human evaluation results show that our model successfully neutralizes or reverses the polarity of news without reducing readability. We release a large annotated dataset that includes both news headlines and full-length articles with polarity labels and meta-data to be used for future research. Our framework has a potential to be used by social scientists, content creators and content consumers in the real world.

Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans

Brain tumors have been one of the most common life-threatening diseases for all mankind. There have beenhuge efforts dedicated to the development of medical imaging techniques and radiomics to diagnose tumor patients quicklyand e?iciently. One of the main aims is to ensure that preoperative overall survival time (OS) prediction is accurate.Recently, deep learning (DL) algorithms, and particularly convolutional neural networks (CNNs) achieved promisingperformances in almost all computer vision fields. CNNs demand large training datasets and high computational costs.However, curating large annotated medical datasets are difficult and resource-intensive. The performances of singlelearners are also unsatisfactory for small datasets. Thus, this study was conducted to improve the performance of CNN models on small volumetric datasets through developing a DL-based ensemble method for OS classification of brain tumorpatients using multimodal magnetic resonance images (MRI). First, we proposed multiview CNNs: OS classifiers basedon representing the 3D MRI data as a set of 2D slices along all three planes (axial, sagittal, and coronal) and processthem using 2D CNNs. Subsequently, the predicted probabilities by the multiview CNN models were fused using standardmachine learning algorithms. The proposed approach was experimentally evaluated on 163 patients obtained from theBraTS?17 training dataset. Our best model achieved an AUC and accuracy values of 0.93 and 92.9%, respectively, onclassifying patients with brain tumors into two OS groups, outperforming current state-of-the-art results. In addition,the FLAIR MRI modality yielded the best classification accuracy compared to other MRI modalities. Similarly, axialprojections had the best classification performance compared to coronal and sagittal projections. Our findings mayprovide valuable insights for physicians in advancing treatment planning via noninvasive and accurate prediction ofsurvival using only MRIs at the time of diagnosis.

Active, continual fine tuning of convolutional neural networks for reducing annotation efforts.

The splendid success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, in medical imaging, it is challenging to create such large annotated datasets, as annotating medical images is not only tedious, laborious, and time consuming, but it also demands costly, specialty-oriented skills, which are not easily accessible. To dramatically reduce annotation cost, this paper presents a novel method to naturally integrate active learning and transfer learning (fine-tuning) into a single framework, which starts directly with a pre-trained CNN to seek "worthy" samples for annotation and gradually enhances the (fine-tuned) CNN via continual fine-tuning. We have evaluated our method using three distinct medical imaging applications, demonstrating that it can reduce annotation efforts by at least half compared with random selection.

Learning from mistakes: Improving spelling correction performance with automatic generation of realistic misspellings

AbstractSequence to sequence models (seq2seq) require a large amount of labelled training data to learn the mapping between the input and output. A large set of misspelled words together with their corrections is needed to train a seq2seq spelling correction system. Low‐resource languages such as Turkish usually lack such large annotated datasets. Although misspelling‐reference pairs can be synthesized with a random procedure, the generated dataset may not well match to genuine human‐made misspellings. This might degrade the performance in realistic test scenarios. In this paper, we propose a novel procedure to automatically introduce human‐like misspellings to legitimate words in Turkish language. Generated human‐like misspellings are used to improve the performance of a seq2seq spelling correction system. The proposed system consists of two separate models; a misspelling generator and a spelling corrector. The generator is trained using a relatively small number of human‐made misspellings and their manual corrections. Reference words and their misspellings are used as inputs and outputs of the generator, respectively. As a result, it is trained to add realistic spelling errors to the valid words. Training data of the spelling corrector is augmented by the generator's human‐like misspellings. In the experiments, we observe that the data augmentation significantly improves the spelling correction performance. Our proposed method yields 5% absolute improvement over the state‐of‐the‐art Turkish spelling correction systems in a test set which contains human‐made misspellings from Twitter messages.