Annotated Images Research Articles

Active Learning Based 3D Semantic Labeling From Images and Videos

3D semantic segmentation is one of the most fundamental problems for 3D scene understanding and has attracted much attention in the field of computer vision. In this paper, we propose an active learning based 3D semantic labeling method for large-scale 3D mesh model generated from images or videos. Taking as input a 3D mesh model reconstructed from the image based 3D modeling system, coupled with the calibrated images, our method outputs a fine 3D semantic mesh model in which each facet is assigned a semantic label. There are three major steps in our framework: 2D semantic segmentation, 2D-3D semantic fusion, and batch image selection. A limited annotation image set is first used to fine-tune a pre-trained semantic segmentation network for obtaining the pixel-wise semantic probability maps. Then all these maps are back-projected into 3D space and fused on the 3D mesh model using Markov Random Field optimization, thus yield a preliminary 3D semantic mesh model and a heat model showing each facet’s confidence. This 3D semantic model is used as a reliable supervisor to select the parts that are not well segmented for manual annotation to boost the performance of the 2D semantic segmentation network, as well as the 3D mesh labeling, in the next iteration. This Training-Fusion-Selection process continues until the label assignment of the 3D mesh model becomes steady. By this means, we significantly reduce the amount for annotation but not the labeling quality of 3D semantic models. Extensive experiments demonstrate the effectiveness and generalization ability of our method on a wide variety of datasets.

Feature Extraction and Selection in Archaeological Images for Automatic Annotation

In this paper, we present two steps in the process of automatic annotation in archeological images. These steps are feature extraction and feature selection. We focus our research on archeological images which are very much studied in our days. It presents the most important steps in the process of automatic annotation in an image. Feature extraction techniques are applied to get the feature that will be used in classifying and recognizing the images. Also, the selection of characteristics reduces the number of unattractive characteristics. However, we reviewed various images of feature extraction techniques to analyze the archaeological images. Each feature represents one or more feature descriptors in the archeological images. We focus on the descriptor shape of the archaeological objects extraction in the images using contour method-based shape recognition of the monuments. So, the feature selection stage serves to acquire the most interesting characteristics to improve the accuracy of the classification. In the feature selection section, we present a comparative study between feature selection techniques. Then we give our proposal of application of methods of selection of the characteristics of the archaeological images. Finally, we calculate the performance of two steps already mentioned: the extraction of characteristics and the selection of characteristics.

A benchmark bone marrow aspirate smear dataset and a multi-scale cell detection model for the diagnosis of hematological disorders.

Research on pathological diagnosis of hematopoietic disorders based on bone marrow aspirate smear images has attracted more and more attention with the development of deep learning methods. However, high quality bone marrow aspirate smear image datasets are not readily available because of the time, the efforts, and the medical knowledge required in the acquisition and manual annotation images. In order to facilitate the research of automated diagnosis of hematological disorders, we constructed a high quality Bone Marrow Aspirate Smear Image Dataset (BMASID), which contains 230 bone marrow aspirate smear images, all with the corresponding labeled images. We used additional clinical images as testing data, which are more challenging because of image noise, cell overlap, cell adhesion, blurred borders of cells and ambiguous types of cells. We also proposed a Cell Recognition Network (CRNet) that was trained on this benchmark dataset. CRNet is comprised of a cell detector to locate and recognize cells in the bone marrow aspirate images, and a cell classifier to classify the types of cells. New anchors and novel evaluation metrics are proposed and applied in CRNet. Benchmark evaluations of the proposed CRNet demonstrated the satisfactory performance of our state-of-the-art methods. Experimental results show that the detection precision by detector is more than 83%, and it is better when compared with other detection methods. After the cell type confirmation by the cell classifier, the precision is more than 95%. Compared with the most popular evaluation metrics Intersection over Union (IoU) and the newly proposed Generalized Intersection over Union (GIoU) used in the object detection benchmarks, our evaluation metrics are more suitable for the cell detection task with ambiguous cell boundaries. The proposed bone marrow aspirate smear image dataset and the proposed evaluation metrics can be used in the training and the evaluation of cell detection models, which contributes to future research in the pathological analysis and auxiliary diagnostic methods of hematological disorders. The codes are available at: https://github.com/SuJie-Med/hematolgical-disorders.

Adding a Visual Communication Tool to the Electronic Health Record to Prevent Pressure Injuries.

Health care organizations no longer receive reimbursement for stage 3, stage 4, and unstageable hospital-acquired pressure injuries, which are never events that require interdisciplinary collaboration to treat and prevent. Perioperative patients are at risk for developing pressure injuries that may not manifest until five or more days after surgery. This quality improvement project aimed to inform inpatient nursing and health care professionals of the potential areas of skin breakdown after surgery using an annotated image (AI) depicting intraoperative patient positioning. To gauge OR nurses' use of the AI, we tracked image addition to the patient records weekly from May 2018 to May 2019 and found an overall average participation rate of 80% to 90%. Adding the AI of surgical positioning and pressure points raised awareness of the potential for skin breakdown and became a valuable communication tool for the inpatient nurses who were providing postoperative care to prevent pressure injuries.

Label cleaning and propagation for improved segmentation performance using fully convolutional networks.

In recent years, fully convolutional networks (FCNs) have been applied to various medical image segmentation tasks. However, it is difficult to generate a large amount of high-quality annotation data to train FCNs for medical image segmentation. Thus, it is desired to achieve high segmentation performances even from incomplete training data. We aim to evaluate performance of FCNs to clean noises and interpolate labels from noisy and sparsely given label images. To evaluate the label cleaning and propagation performance of FCNs, we used 2D and 3D FCNs to perform volumetric brain segmentation from magnetic resonance image volumes, based on network training on incomplete training datasets from noisy and sparse annotation. The experimental results using pseudo-incomplete training data showed that both 2D and 3D FCNs could provide improved segmentation results from the incomplete training data, especially by using three orthogonal annotation images for network training. This paper presented a validation for label cleaning and propagation based on FCNs. FCNs might have the potential to achieve improved segmentation performances even from sparse annotation data including possible noises by manual annotation, which can be an important clue to more efficient annotation.

Performance of an artificial intelligence tool with real-time clinical workflow integration - Detection of intracranial hemorrhage and pulmonary embolism.

Performance of an artificial intelligence tool with real-time clinical workflow integration - Detection of intracranial hemorrhage and pulmonary embolism.

Self-Attentive Spatial Adaptive Normalization for Cross-Modality Domain Adaptation.

Despite the successes of deep neural networks on many challenging vision tasks, they often fail to generalize to new test domains that are not distributed identically to the training data. The domain adaptation becomes more challenging for cross-modality medical data with a notable domain shift. Given that specific annotated imaging modalities may not be accessible nor complete. Our proposed solution is based on the cross-modality synthesis of medical images to reduce the costly annotation burden by radiologists and bridge the domain gap in radiological images. We present a novel approach for image-to-image translation in medical images, capable of supervised or unsupervised (unpaired image data) setups. Built upon adversarial training, we propose a learnable self-attentive spatial normalization of the deep convolutional generator network's intermediate activations. Unlike previous attention-based image-to-image translation approaches, which are either domain-specific or require distortion of the source domain's structures, we unearth the importance of the auxiliary semantic information to handle the geometric changes and preserve anatomical structures during image translation. We achieve superior results for cross-modality segmentation between unpaired MRI and CT data for multi-modality whole heart and multi-modal brain tumor MRI (T1/T2) datasets compared to the state-of-the-art methods. We also observe encouraging results in cross-modality conversion for paired MRI and CT images on a brain dataset. Furthermore, a detailed analysis of the cross-modality image translation, thorough ablation studies confirm our proposed method's efficacy.

Few-shot prototype alignment regularization network for document image layout segementation

Despite the great performance in layout analysis tasks made by semantic segmentation, they usually need a large number of annotated images for training and are difficult to learn a new category which is absent in the training categories. Meta-learning and few-shot segmentation have been developed to solve the above two difficulties. In this paper, we propose a novel method dubbed Few-Shot Prototype Alignment Regularization Network (FS-PARN). The FS-PARN method is inspired by recent studies in both metric learning and few-shot segmentation, which just need a few annotated images to solve the above two difficulties. Our FS-PARN method can make better use of the information of the support set by metric learning and have a better effect on image segmentation. It learns classification prototype within an embedding space and then completes pixel classification by matching each pixel on the query image with the learned prototype. In addition to obtaining high-quality prototypes through metric learning methods, our FS-PARN method also introduces prototype alignment regularization between support and query sets to make segmentation better. Notably, our FS-PARN model achieves the mean-IoU score of 28.8% and 31.7% on the practical document image datasets, i.e. PASCAL-5i, DSSE-200, and Layout Analysis Dataset, for 1-shot and 5-shot settings respectively.

A deep learning approach using synthetic images for segmenting and estimating 3D orientation of nanoparticles in EM images

Background and objective: Nanoparticles present properties that can be applied to a wide range of fields such as biomedicine, electronics or optics. The type of properties depends on several characteristics, being some of them related with the particle structure. A proper characterization of nanoparticles is crucial since it could affect their applications. To characterize a particle shape and size, the nanotechnologists employ Electron Microscopy (EM) to obtain images of nanoparticles and perform measures over them. This task could be tedious, repetitive and slow, we present a Deep Learning method based on Convolutional Neural Networks (CNNs) to detect, segment, infer orientations and reconstruct microscope images of nanoparticles. Since machine learning algorithms depend on annotated data and there is a lack of annotated datasets of nanoparticles, our work makes use of artificial datasets of images resembling real nanoparticles photographs.Methods: Our work is divided into three tasks. Firstly, a method to create annotated datasets of artificial images resembling Scanning Electron Microscope (SEM). Secondly, two models of convolutional neural networks are trained using the artificial datasets previously generated, the first one is in charge of the detection and segmentation of the nanoparticles while the second one will infer the nanoparticle orientation. Finally, the 3D reconstruction module will recreate in a 3D scene the set of detected particles.Results: We have tested our method with five different shapes of basic nanoparticles: spheres, cubes, ellipsoids, hexagonal discs and octahedrons. An analysis of the reconstructions was conducted by manually comparing each of them with the real images. The results obtained have been promising, the particles are segmented and reconstructed accordingly to their shapes and orientations.Conclusions: We have developed a method for nanoparticle detection and segmentation in microscope images. Moreover, we can also infer an approximation of the 3D orientation of the particles and, in conjunction with the detections, create a 3D reconstruction of the photographs. The novelty of our approximation lies in the dataset used. Instead of using annotated images, we have created the datasets simulating the microscope images by using basic geometrical objects that imitate real nanoparticles.

DramaQA: 계층적 질의응답과 함께하는 등장인물 중심 비디오 스토리 이해

Despite recent progress on computer vision and natural language processing, developing a machine that can understand video story is still hard to achieve due to the intrinsic difficulty of video story. Moreover, researches on how to evaluate the degree of video understanding based on human cognitive process have not progressed as yet. In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focuses on two perspectives: 1) Hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) Character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama Another Miss Oh and it contains 17,983 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered annotations, including visual bounding boxes, behaviors and emotions of main characters, and coreference resolved scripts. Additionally, we suggest Multi-level Context Matching model which hierarchically understands character-centered representations of video to answer questions. We release our dataset and model publicly for research purposes, and we expect our work to provide a new perspective on video story understanding research.

Active Cell Appearance Model Induced Generative Adversarial Networks for Annotation-Efficient Cell Segmentation and Identification on Adaptive Optics Retinal Images.

Data annotation is a fundamental precursor for establishing large training sets to effectively apply deep learning methods to medical image analysis. For cell segmentation, obtaining high quality annotations is an expensive process that usually requires manual grading by experts. This work introduces an approach to efficiently generate annotated images, called "A-GANs", created by combining an active cell appearance model (ACAM) with conditional generative adversarial networks (C-GANs). ACAM is a statistical model that captures a realistic range of cell characteristics and is used to ensure that the image statistics of generated cells are guided by real data. C-GANs utilize cell contours generated by ACAM to produce cells that match input contours. By pairing ACAM-generated contours with A-GANs-based generated images, high quality annotated images can be efficiently generated. Experimental results on adaptive optics (AO) retinal images showed that A-GANs robustly synthesize realistic, artificial images whose cell distributions are exquisitely specified by ACAM. The cell segmentation performance using as few as 64 manually-annotated real AO images combined with 248 artificially-generated images from A-GANs was similar to the case of using 248 manually-annotated real images alone (Dice coefficients of 88% for both). Finally, application to rare diseases in which images exhibit never-seen characteristics demonstrated improvements in cell segmentation without the need for incorporating manual annotations from these new retinal images. Overall, A-GANs introduce a methodology for generating high quality annotated data that statistically captures the characteristics of any desired dataset and can be used to more efficiently train deep-learning-based medical image analysis applications.

A dataset and a methodology for intraoperative computer-aided diagnosis of a metastatic colon cancer in a liver

The lack of pixel-wise annotated images severely hinders the deep learning approach to computer-aided diagnosis in histopathology. This research creates a public database comprised of: (i) a dataset of 82 histopathological images of hematoxylin-eosin stained frozen sections acquired intraoperatively on 19 patients diagnosed with metastatic colon cancer in a liver; (ii) corresponding pixel-wise ground truth maps annotated by four pathologists, two residents in pathology, and one final-year student of medicine. The Fleiss' kappa equal to 0.74 indicates substantial inter-annotator agreement; (iii) two datasets with images stain-normalized relative to two target images; (iv) development of two conventional machine learning and three deep learning-based diagnostic models. The database is available at http://cocahis.irb.hr. For binary, cancer vs. non-cancer, pixel-wise diagnosis we develop: SVM, kNN, U-Net, U-Net++, and DeepLabv3 classifiers that combine results from original images and stain-normalized images, which can be interpreted as different views. On average, deep learning classifiers outperformed SVM and kNN classifiers on an independent test set 14% in terms of micro balanced accuracy, 15% in terms of the micro F1 score, and 26% in terms of micro precision. As opposed to that, the difference in performance between deep classifiers is within 2%. We found an insignificant difference in performance between deep classifiers trained from scratch and corresponding classifiers pre-trained on non-domain image datasets. The best micro balanced accuracy estimated on the independent test set by the U-Net++ classifier equals 89.34%. Corresponding amounts of F1 score and precision are, respectively, 83.67% and 81.11%.

Automatic Multiclass Instance Segmentation of Concrete Damage Using Deep Learning Model

Concrete is one of the primary and most commonly used materials for the construction of buildings, roads, bridges, and dams. But it may lose its strength with age, moisture content, or due to other factors. As a result, little damage (crack and spall) to these structures may lead to sudden collapse or breakage, which in turn can decimate many lives with economic losses. Hence, to ensure their strength with the proper load, accurate screening of concrete surface damage is necessary for maintenance engineers to understand and evaluate the severity of the damage. Few preceding vision-based studies have proposed object detection and semantic segmentation approaches to carry out damage detection; however, the developed models could not segment different objects of the same class. The semantic segmentation approach separates regions that contain only objects of the same class. Concrete surface damage, on the other hand, may contain multiple objects, such as the Horizontal Crack, Vertical Crack, Diagonal Crack, Branch Crack, and Spall, that need to be segmented separately so that specific measures can be taken. This study utilizes Mask Region-Based Convolutional Neural Network (Mask R-CNN) to manage concrete damage images and employ it to detect and segment defects in civil infrastructure with multiple objects. As we understand, this is the first research involving a Structural Health Monitoring (SHM) application for damage detection using instance segmentation. Our proposed model has been trained and tested on a dataset containing 800 images (480*480 pixels) of different types of crack and spall are collected from distinct structures of CSIR-CEERI, Pilani, campus. To test and validate the generosity, an additional 96 damage images are downloaded using Google. The pre-processed and annotated images are used to train and validate the Mask R-CNN Classifier. Our experimental results show that damage can be classified efficiently with 95.13% accuracy on a custom dataset and 96.87% on randomly picked images.

Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

A Deep Learning Pipeline for Nucleus Segmentation.

Deep learning is rapidly becoming the technique of choice for automated segmentation of nuclei in biological image analysis workflows. In order to evaluate the feasibility of training nuclear segmentation models on small, custom annotated image datasets that have been augmented, we have designed a computational pipeline to systematically compare different nuclear segmentation model architectures and model training strategies. Using this approach, we demonstrate that transfer learning and tuning of training parameters, such as the composition, size, and preprocessing of the training image dataset, can lead to robust nuclear segmentation models, which match, and often exceed, the performance of existing, off-the-shelf deep learning models pretrained on large image datasets. We envision a practical scenario where deep learning nuclear segmentation models trained in this way can be shared across a laboratory, facility, or institution, and continuously improved by training them on progressively larger and varied image datasets. Our work provides computational tools and a practical framework for deep learning-based biological image segmentation using small annotated image datasets. Published [2020]. This article is a U.S. Government work and is in the public domain in the USA.

Active learning for image preparation of automatic vending machine (AVM) employing transfer learning method

In this paper, we employed active learning methods to prepare annotated images for our training of automatic vending machine (AVM) system, in order to minimize human annotation cost. Due to the tiny data of our system, transfer learning approach is used by implementing the already trained Yolov3-tiny model for COCO dataset as our training start. Also, we evaluated the effectiveness of 3 annotation strategies: smallest annotation area (SAA), largest annotation area (LAA) and moderate annotation area (MAA), for photos of top views from above. The results show that the idea of employing active learning methods to prepare annotated data is feasible. Also, the annotation strategy of MAA demonstrates the superior performance, for its enough object area and the least background area.

RETRACTED: Sports image detection based on particle swarm optimization algorithm

Athletes can still insist on the practicality of sports practice in today's sports environment by locating image detection technology, video and video files, as well as broadcast data detection key format. However, from the most advanced of the situation in the country, goal can confirm that the light detection dong area is still immature. So let's take a look at the video cloth applied to detect motion pictures in ages to search for software. Method are looking at the side of detection, grayscale processing, grasping the project, target identification, and see the light detection era. The actual sports programs and the desire to meet the different needs of the sports image detection mix. At the same time, in the present study, found that the popularity of athletes and judge, motion recognition of motor behavior. Along with affirmed the effectiveness of this method of research, have created a platform for inspection. Results show that the research method is always practical, can provide a theoretical basis for future research. A set of rules contained in a hierarchy: First, the Particle Swarm Optimize [PSO] algorithm is used to the annotation image per day. In the experiment, 3 sets of data were used to evaluate a set of rules, the results show that the performance of the algorithm.

Biomedical image classification made easier thanks to transfer and semi-supervised learning

Background and objectivesDeep learning techniques are the state-of-the-art approach to solve image classification problems in biomedicine; however, they require the acquisition and annotation of a considerable volume of images. In addition, using deep learning libraries and tuning the hyperparameters of the networks trained with them might be challenging for several users. These drawbacks prevent the adoption of these techniques outside the machine-learning community. In this work, we present an Automated Machine Learning (AutoML) method to deal with these problems. MethodsOur AutoML method combines transfer learning with a new semi-supervised learning procedure to train models when few annotated images are available. In order to facilitate the dissemination of our method, we have implemented it as an open-source tool called ATLASS. Finally, we have evaluated our method with two benchmarks of biomedical image classification datasets. ResultsOur method has been thoroughly tested both with small datasets and partially annotated biomedical datasets; and, it outperforms, both in terms of speed and accuracy, the existing AutoML tools when working with small datasets; and, might improve the accuracy of models up to a 10% when working with partially annotated datasets. ConclusionsThe work presented in this paper allows the use of deep learning techniques to solve an image classification problem with few resources. Namely, it is possible to train deep models with small, and partially annotated datasets of images. In addition, we have proven that our AutoML method outperforms other AutoML tools both in terms of accuracy and speed when working with small datasets.

POLLEN73S: An image dataset for pollen grains classification

The classification of pollen species and types is an important task in many areas like forensic palynology, archaeological palynology, and melissopalynology. This paper presents a new public annotated image dataset for the Brazilian Savanna called POLLEN73S composed of 2523 images from 73 pollen types. Using the state-of-the-art Convolutional Neural Networks (CNNs), we provide a baseline for pollen grain classification. Our experiments showed evidence that DenseNet-201 and ResNet-50 have superior performance against the other CNNs tested, achieving precision results of 95.7% and 94.0%, respectively. Due to its category coverage and satisfactory diversity of examples, POLLEN73S offers a diversity of pollen grain to guide progress in computer vision to solve Palynology problems.

The image annotation algorithm using convolutional features from intermediate layer of deep learning

The automatic image annotation is an effective computer operation that predicts the annotation of an unknown image by automatically learning potential relationships between the semantic concept space and the visual feature space in the annotation image dataset. Usually, the auto-labeling image includes the processing: learning processing and labeling processing. Existing image annotation methods that employ convolutional features of deep learning methods have a number of limitations, including complex training and high space/time expenses associated with the image annotation procedure. Accordingly, this paper proposes an innovative method in which the visual features of the image are presented by the intermediate layer features of deep learning, while semantic concepts are represented by mean vectors of positive samples. Firstly, the convolutional result is directly output in the form of low-level visual features through the mid-level of the pre-trained deep learning model, with the image being represented by sparse coding. Secondly, the positive mean vector method is used to construct visual feature vectors for each text vocabulary item, so that a visual feature vector database is created. Finally, the visual feature vector similarity between the testing image and all text vocabulary is calculated, and the vocabulary with the largest similarity used for annotation. Experiments on the datasets demonstrate the effectiveness of the proposed method; in terms of F1 score, the proposed method’s performance on the Corel5k dataset and IAPR TC-12 dataset is superior to that of MBRM, JEC-AF, JEC-DF, and 2PKNN with end-to-end deep features.