Image Tagging Research Articles

Integration of Three-dimensional Magnetic Resonance Imaging Spectroscopy with the Leksell GammaPlan Radiosurgical Planning Station for the Treatment of Brain Tumors.

IntroductionMRI multivoxel spectroscopy mapping is helpful in surgical decision-making. Unfortunately, in daily practice, MRI multivoxel spectroscopy mapping is not always compatible with the current version of Leksell GammaPlan (LGP) (Elekta, Stockholm, Sweden). The aim of this study is to develop a tool to allow the use of this modality in radiosurgical treatments using LGP.Material and methodsMultivoxel spectroscopy digital imaging and communications in medicine (DICOM) images were analyzed to identify tags to be modified to make the images compatible with LGP. We identify four important tags to be modified for compatibility with LGP. Using Python language, a new software was designed to modify the identified tags and allow the automatic conversion of images to meet LGP requirements.ResultsBy modifying the tags of DICOM images, we could use spectroscopic cartography images in radiosurgical planning using LGP. We created a software to reproduce these modifications using a simple and rapid interface. This software executes all the protocols established in the methodology.ConclusionThe new software, “GP Adapting Solution”, can convert any DICOM image and make it compatible with LGP. The integration of multivoxel spectroscopic images was feasible and could be used for radiosurgical planning. This work is the first step in allowing the potential use of new MRI modalities in radiosurgical planning using LGP. The next steps are to evaluate the impact of these modalities in radiosurgical treatments and to develop methods for integrating other imaging modalities.

CNN based Image Classification Model

Images are the fastest growing content, they contribute significantly to the amount of data generated on the internet every day. Image classification is a challenging problem that social media companies work on vigorously to enhance the user’s experience with the interface. The recent advances in the field of machine learning and computer vision enables personalized suggestions and automatic tagging of images. Convolutional neural network is a hot research topic these days in the field of machine learning. With the help of immensely dense labelled data available on the internet the networks can be trained to recognize the differentiating features among images under the same label. New neural network algorithms are developed frequently that outperform the state-of-art machine learning algorithms. Recent algorithms have managed to produce error rates as low as 3.1%. In this paper the architecture of important CNN algorithms that have gained attention are discussed, analyzed and compared and the concept of transfer learning is used to classify different breeds of dogs..

Improved tagged cardiac MRI myocardium strain analysis by leveraging cine segmentation

Background and objectivesTagged MR images provide an effective way for regional analysis of the myocardium strain. A reliable myocardium strain analysis requires both correct segmentation and accurate motion tracking of the myocardium during the cardiac cycle. While many algorithms have been proposed for accurate tracking of the myocardium in tagged MR images, little focus has been placed on ensuring correct segmentation of the tagged myocardium during the cardiac cycle. Myocardial strain analysis is usually done by segmenting the myocardium in end-diastole, generating a mesh from the segmentation, propagating the mesh through the cardiac cycle using the output deformation field from motion tracking, and measuring strain on the deforming mesh. Due to the imposed tag strips on the anatomy, identification of the myocardium boundaries is challenging in tagged MR images. As a result, there is no guarantee that the propagated mesh is annotating the myocardium accurately through the cardiac cycle. Moreover, clinical studies indicate that incorrect myocardium annotation can result in overestimation of myocardial strains. MethodsWe introduce a method to improve reliability of strain analysis by proposing a mesh which correctly segments the myocardium in tagged MRI by leveraging the available cine MRI segmentation. In particular, we generate a series of mesh proposals using the cine MRI segmentation and find the propagated mesh proposal which gives the most accurate full-cycle myocardium segmentation. ResultsThe mesh selection algorithm was tested on 22 2D MRI scans of diseased and healthy hearts. The proposed algorithm provided more accurate whole-cycle myocardium segmentation compared to the propagated end-diastolic mesh. Regional myocardium strain was measured for 10 3D MRI scans of healthy volunteers using the proposed mesh and the end-diastolic mesh. The measured strain using the proposed mesh was more similar to the expected myocardium strain for a healthy heart than the measured strain using the end-diastolic mesh. ConclusionThe proposed approach provides accurate whole-cycle tagged myocardium segmentation and more reliable myocardium strain analysis.

A Latent Semantic Analysis Based Image Tag Optimization Method

According to the nominal tags, the adverb and adjective high-level semantic tags which is described human emotion are easily to exceed the handling scope. Furthermore, the majority of optimisation methods exists many problems (e.g., although inputting a high application cost, it cannot get a satisfactory effect). This paper proposed an effective image tag optimisation algorithm which was composed of two important parts: firstly, a random walk model was used to process the initial image tag information, the image relationship diagram and the tag relationship diagram were structured based on image vision similarity and tag relevancy respectively, then, the image and tag information was spread through a dual-diagram (image relationship diagram and tag relationship diagram) walk mode randomly; secondly, a new image tag optimisation model was built according to three factors, i.e., semantic uniformity, noise sparsity and result matrix sparsity. In the experimental stage, the effectiveness of this model and the superiority of pre-processing were verified by experiment. Compared to other methods, the experimental result indicates that this algorithm is more reasonable and efficient.

COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval

This paper contributes to cross-lingual image annotation and retrieval in terms of data and baseline methods. We propose COCO-CN, a novel dataset enriching MS-COCO with manually written Chinese sentences and tags. For more effective annotation acquisition, we develop a recommendation-assisted collective annotation system, automatically providing an annotator with several tags and sentences deemed to be relevant with respect to the pictorial content. Having 20,342 images annotated with 27,218 Chinese sentences and 70,993 tags, COCO-CN is currently the largest Chinese-English dataset that provides a unified and challenging platform for cross-lingual image tagging, captioning and retrieval. We develop conceptually simple yet effective methods per task for learning from cross-lingual resources. Extensive experiments on the three tasks justify the viability of the proposed dataset and methods. Data and code are publicly available at https://github.com/li-xirong/coco-cn

Enhancing the Quality of Image Tagging Using a Visio-Textual Knowledge Base

Auto-tagging of images is important for image understanding and for tag-based applications viz. image retrieval, visual question-answering, image captioning, etc. Although existing tagging methods incorporate both visual and textual information to assign/refine tags, they lag in tag-image relevance, completeness, and preciseness, thereby resulting in the unsatisfactory performance of tag-based applications. In order to bridge this gap, we propose a novel framework for tag assignment using knowledge embedding (TAKE) from a proposed external knowledge base, considering properties such as Rarity, Newness, Generality, and Naturalness ( RNGN properties). These properties help in providing a rich semantic representation to images. Existing knowledge bases provide multiple types of relations extracted through only one modality, either text or visual, which is not effective in image related applications. We construct a simple yet effective Visio-Textual Knowledge Base (VTKB) with only four relations using reliable resources such as Wikipedia, thesauruses, dictionaries, etc. Our large scale experiments demonstrate that the proposed combination of TAKE and VTKB assigns a large number of high quality tags in comparison to the ConceptNet and ImageNet knowledge bases when used in conjunction with TAKE. Also, the effectiveness of knowledge embedding through VTKB is evaluated for image tagging and tag-based image retrieval (TBIR).

MaizeDIG: Maize Database of Images and Genomes.

Background: An organism can be described by its observable features (phenotypes) and the genes and genomic information (genotypes) that cause these phenotypes. For many decades, researchers have tried to find relationships between genotypes and phenotypes, and great strides have been made. However, improved methods and tools for discovering and visualizing these phenotypic relationships are still needed. The maize genetics and genomics database (MaizeGDB, www.maizegdb.org) provides an array of useful resources for diverse data types including thousands of images related to mutant phenotypes in Zea mays ssp. mays (maize). To integrate mutant phenotype images with genomics information, we implemented and enhanced the web-based software package BioDIG (Biological Database of Images and Genomes). Findings: We developed a genotype-phenotype database for maize called MaizeDIG. MaizeDIG has several enhancements over the original BioDIG package. MaizeDIG, which supports multiple reference genome assemblies, is seamlessly integrated with genome browsers to accommodate custom tracks showing tagged mutant phenotypes images in their genomic context and allows for custom tagging of images to highlight the phenotype. This is accomplished through an updated interface allowing users to create image-to-gene links and is accessible via the image search tool. Conclusions: We have created a user-friendly and extensible web-based resource called MaizeDIG. MaizeDIG is preloaded with 2,396 images that are available on genome browsers for 10 different maize reference genomes. Approximately 90 images of classically defined maize genes have been manually annotated. MaizeDIG is available at http://maizedig.maizegdb.org/. The code is free and open source and can be found at https://github.com/Maize-Genetics-and-Genomics-Database/maizedig.

Blending Ensemble of Fine-Tuned Convolutional Neural Networks Applied to Mammary Image Classification

Medical images classification is a challenging research topic in the field of computer vision, especially when applied to diagnosis of breast cancer (BC). Nowadays, histopathological image is marked as the gold standard for diagnosing BC. However, such diagnosis is heavily dependent on the clinician's experience, which is extremely time consuming and is subjected to human error even for experienced doctors. To address those problems, this paper implements an automated method for distinguishing the benign from the malignant tumor based on a convolutional neural network (CNN). Traditional deep CNN and machine learning methods not only lead to poor performance, but also fail to make full use of the long-term dependence between some key features and image tags. To further meet the high accuracy requirement of diagnosis, according to the characteristics of histopathological images, we propose a novel CNN framework. Firstly, a normal image is augmented to solve the problem about having a limited database. Secondly, we introduce transfer learning to obtain more accurate weight parameters that were pre-trained on the ImageNet. Thirdly, we combine various features extracted by many individual models to obtain comprehensive features. Finally, random forest is introduced to enforce classification. The experimental results show that novel CNN frameworks have better performance compared with individual models, including DenseNet and ResNet. Experimental results are able to prove the effectiveness of our strategy.

Privacy-aware Tag Recommendation for Accurate Image Privacy Prediction

Online images’ tags are very important for indexing, sharing, and searching of images, as well as surfacing images with private or sensitive content, which needs to be protected. Social media sites such as Flickr generate these metadata from user-contributed tags. However, as the tags are at the sole discretion of users, these tags tend to be noisy and incomplete. In this article, we present a privacy-aware approach to automatic image tagging, which aims at improving the quality of user annotations, while also preserving the images’ original privacy sharing patterns. Precisely, we recommend potential tags for each target image by mining privacy-aware tags from the most similar images of the target image, which are obtained from a large collection. Experimental results show that, although the user-input tags compose noise, our privacy-aware approach is able to predict accurate tags that can improve the performance of a downstream application on image privacy prediction and outperforms an existing privacy-oblivious approach to image tagging. The results also show that, even for images that do not have any user tags, our proposed approach can recommend accurate tags. Crowd-sourcing the predicted tags exhibits the quality of our privacy-aware recommended tags. Our code, features, and the dataset used in experiments are available at: https://github.com/ashwinitonge/privacy-aware-tag-rec.git.

Greater good, empowerment and democratization? Affordances of the crowdsourcing transcription projects

Digital technology and Internet access have created new possibilities for museums and archives for digitization of their collections. Steadily, more museums are experimenting with inviting their audiences to participate in tagging images, annotating, transcribing historical texts or cropping photographs. This article is an exploration of visual and functional aspects of various digital interfaces frequently being used in crowdsourcing projects involving transcribing manuscripts. The empirical material has been collected through interviews with the editors of the projects and systematic technical walkthroughs of MediaWiki platforms (Edvard Munch’s Writings and Transcribe Bentham) and Zooniverse platforms (AnnoTate and Shakespeare’s World). The analysis aims to explore platforms’ affordances (Gibson 1978), in other words the opportunities that the layout and design offer to users interacting with facsimiles of manuscripts (‘digital networked objects’) (Cameron and Mengler 2015). The questions raised are whether and how the interfaces empower users and perform as a democratic actor providing the volunteers with agency. The platforms’ interfaces have emerged as an important and undervalued actor-network of elements which configure heterogeneous relations among actors and influence users’ engagement.

Dynamically Identifying Deep Multimodal Features for Image Privacy Prediction

With millions of images shared online, privacy concerns are on the rise. In this paper, we propose an approach to image privacy prediction by dynamically identifying powerful features corresponding to objects, scene context, and image tags derived from Convolutional Neural Networks for each test image. Specifically, our approach identifies the set of most “competent” features on the fly, according to each test image whose privacy has to be predicted. Experimental results on thousands of Flickr images show that our approach predicts the sensitive (or private) content more accurately than the models trained on each individual feature set (object, scene, and tags alone) or their combination.

Deep0Tag: Deep Multiple Instance Learning for Zero-Shot Image Tagging

Zero-shot learning aims to perform visual reasoning about unseen objects. In-line with the success of deep learning on object recognition problems, several end-to-end deep models for zero-shot recognition have been proposed in the literature. These models are successful in predicting a single unseen label given an input image but do not scale to cases where multiple unseen objects are present. Here, we focus on the challenging problem of zero-shot image tagging, where multiple labels are assigned to an image, that may relate to objects, attributes, actions, events, and scene type. Discovery of these scene concepts requires the ability to process multi-scale information. To encompass global as well as local image details, we propose an automatic approach to locate relevant image patches and model image tagging within the Multiple Instance Learning (MIL) framework. To the best of our knowledge, we propose the first end-to-end trainable deep MIL framework for the multi-label zero-shot tagging problem. We explore several alternatives for instance-level evidence aggregation and perform an extensive ablation study to identify the optimal pooling strategy. Due to its novel design, the proposed framework has several interesting features: 1) unlike previous deep MIL models, it does not use any off-line procedure (e.g., Selective Search or EdgeBoxes) for bag generation. 2) During test time, it can process any number of unseen labels given their semantic embedding vectors. 3) Using only image-level seen labels as weak annotation, it can produce a localized bounding box for each predicted label. We experiment with the large-scale NUS-WIDE and MS-COCO datasets and achieve superior performance across conventional, zero-shot, and generalized zero-shot tagging tasks.

Fairness in Proprietary Image Tagging Algorithms: A Cross-Platform Audit on People Images

There are increasing expectations that algorithms should behave in a manner that is socially just. We consider the case of image tagging APIs and their interpretations of people images. Image taggers have become indispensable in our information ecosystem, facilitating new modes of visual communication and sharing. Recently, they have become widely available as Cognitive Services. But while tagging APIs offer developers an inexpensive and convenient means to add functionality to their creations, most are opaque and proprietary. Through a cross-platform comparison of six taggers, we show that behaviors differ significantly. While some offer more interpretation on images, they may exhibit less fairness toward the depicted persons, by misuse of gender-related tags and/or making judgments on a person’s physical appearance. We also discuss the difficulties of studying fairness in situations where algorithmic systems cannot be benchmarked against a ground truth.

Social B(eye)as: Human and Machine Descriptions of People Images

Image analysis algorithms have become an indispensable tool in our information ecosystem, facilitating new forms of visual communication and information sharing. At the same time, they enable large-scale socio-technical research which would otherwise be difficult to carry out. However, their outputs may exhibit social bias, especially when analyzing people images. Since most algorithms are proprietary and opaque, we propose a method of auditing their outputs for social biases. To be able to compare how algorithms interpret a controlled set of people images, we collected descriptions across six image tagging algorithms. In order to compare these results to human behavior, we also collected descriptions on the same images from crowdworkers in two anglophone regions. The dataset we present consists of tags from these eight taggers, along with a typology of concepts, and a python script to calculate vector scores for each image and tagger. Using our methodology, researchers can see the behaviors of the image tagging algorithms and compare them to those of crowdworkers. Beyond computer vision auditing, the dataset of humanand machine-produced tags, the typology, and the vectorization method can be used to explore a range of research questions related to both algorithmic and human behaviors.

Atlas-Based Tongue Muscle Correlation Analysis From Tagged and High-Resolution Magnetic Resonance Imaging.

Purpose Intrinsic and extrinsic tongue muscles in healthy and diseased populations vary both in their intra- and intersubject behaviors during speech. Identifying coordination patterns among various tongue muscles can provide insights into speech motor control and help in developing new therapeutic and rehabilitative strategies. Method We present a method to analyze multisubject tongue muscle correlation using motion patterns in speech sound production. Motion of muscles is captured using tagged magnetic resonance imaging and computed using a phase-based deformation extraction algorithm. After being assembled in a common atlas space, motions from multiple subjects are extracted at each individual muscle location based on a manually labeled mask using high-resolution magnetic resonance imaging and a vocal tract atlas. Motion correlation between each muscle pair is computed within each labeled region. The analysis is performed on a population of 16 control subjects and 3 post-partial glossectomy patients. Results The floor-of-mouth (FOM) muscles show reduced correlation comparing to the internal tongue muscles. Patients present a higher amount of overall correlation between all muscles and exercise en bloc movements. Conclusions Correlation matrices in the atlas space show the coordination of tongue muscles in speech sound production. The FOM muscles are weakly correlated with the internal tongue muscles. Patients tend to use FOM muscles more than controls to compensate for their postsurgery function loss.

Object Color Estimation with Dominant Color in Automated Image Semantic Description Generation Tasks

The automated image tagging is an important part of modern search engines. The generated image tags can be constructed from object names and their attributes, for example, colors. This work presents an object color name detection real-time algorithm. It is applicable to any automatic object detection and localization systems. The presented algorithm is fast enough to run after the existing real-time object detection system, without adding visible overhead. The algorithm uses k-means to detect the dominant color and selects the correct name for the color via Delta E (CIE 2000).

Filtering Instagram Hashtags Through Crowdtagging and the HITS Algorithm

Instagram is a rich source for mining descriptive tags for images and multimedia in general. The tags–image pairs can be used to train automatic image annotation (AIA) systems in accordance with the learning by example paradigm. In previous studies, we had concluded that, on average, 20% of the Instagram hashtags are related to the actual visual content of the image they accompany, i.e., they are descriptive hashtags, while there are many irrelevant hashtags, i.e., stop-hashtags, that are used across totally different images just for gathering clicks and for searchability enhancement. In this paper, we present a novel methodology, based on the principles of collective intelligence that helps in locating those hashtags. In particular, we show that the application of a modified version of the well-known hyperlink-induced topic search (HITS) algorithm, in a crowdtagging context, provides an effective and consistent way for finding pairs of Instagram images and hashtags, which lead to representative and noise-free training sets for content-based image retrieval. As a proof of concept, we used the crowdsourcing platform Figure-eight to allow collective intelligence to be gathered in the form of tag selection (crowdtagging) for Instagram hashtags. The crowdtagging data of Figure-eight are used to form bipartite graphs in which the first type of nodes corresponds to the annotators and the second type to the hashtags they selected. The HITS algorithm is first used to rank the annotators in terms of their effectiveness in the crowdtagging task and then to identify the right hashtags per image.

A probabilistic topic model for event-based image classification and multi-label annotation

We propose an enhanced latent topic model based on latent Dirichlet allocation and convolutional neural nets for event classification and annotation in images. Our model builds on the semantic structure relating events, objects and scenes in images. Based on initial labels extracted from convolution neural networks (CNNs), and possibly user-defined tags, we estimate the event category and final annotation of an image through a refinement process based on the expectation–maximization (EM) algorithm. The EM steps allow to progressively ascertain the class category and refine the final annotation of the image. Our model can be thought of as a two-level annotation system, where the first level derives the image event from CNN labels and image tags and the second level derives the final annotation consisting of event-related objects/scenes. Experimental results show that the proposed model yields better classification and annotation performance in the two standard datasets: UIUC-Sports and WIDER.

The Role of Social Interactions in Motor Performance: Feasibility Study Toward Enhanced Motivation in Telerehabilitation.

BackgroundRobot-mediated telerehabilitation has the potential to provide patient-tailored cost-effective rehabilitation. However, compliance with therapy can be a problem that undermines the prospective advantages of telerehabilitation technologies. Lack of motivation has been identified as a major factor that hampers compliance. Exploring various motivational interventions, the integration of citizen science activities in robotics-based rehabilitation has been shown to increase patients’ motivation to engage in otherwise tedious exercises by tapping into a vast array of intrinsic motivational drivers. Patient engagement can be further enhanced by the incorporation of social interactions.ObjectiveHerein, we explored the possibility of bolstering engagement in physical therapy by leveraging cooperation among users in an environmental citizen science project. Specifically, we studied how the integration of cooperation into citizen science influences user engagement, enjoyment, and motor performance. Furthermore, we investigated how the degree of interdependence among users, such that is imposed through independent or joint termination (JT), affects participation in citizen science-based telerehabilitation.MethodsWe developed a Web-based citizen science platform in which users work in pairs to classify images collected by an aquatic robot in a polluted water canal. The classification was carried out by labeling objects that appear in the images and trashing irrelevant labels. The system was interfaced by a haptic device for fine motor rehabilitation. We recruited 120 healthy volunteers to operate the platform. Of these volunteers, 98 were cooperating in pairs, with 1 user tagging images and the other trashing labels. The other 22 volunteers performed both tasks alone. To vary the degree of interdependence within cooperation, we implemented independent and JTs.ResultsWe found that users’ engagement and motor performance are modulated by their assigned task and the degree of interdependence. Motor performance increased when users were subjected to independent termination (P=.02), yet enjoyment decreased when users were subjected to JT (P=.005). A significant interaction between the type of termination and the task was found to influence productivity (P<.001) as well as mean speed, peak speed, and path length of the controller (P=.01, P=.006, and P<.001, respectively).ConclusionsDepending on the type of termination, cooperation was not always positively associated with engagement, enjoyment, and motor performance. Therefore, enhancing user engagement, satisfaction, and motor performance through cooperative citizen science tasks relies on both the degree of interdependence among users and the perceived nature of the task. Cooperative citizen science may enhance motivation in robotics-based telerehabilitation, if designed attentively.

Social Anchor-Unit Graph Regularized Tensor Completion for Large-Scale Image Retagging.

Image retagging aims to improve the tag quality of social images by completing the missing tags, rectifying the noise-corrupted tags, and assigning new high-quality tags. Recent approaches simultaneously explore visual, user and tag information to improve the performance of image retagging by mining the tag-image-user associations. However, such methods will become computationally infeasible with the rapidly increasing number of images, tags and users. It has been proven that the anchor graph can significantly accelerate large-scale graph-based learning by exploring only a small number of anchor points. Inspired by this, we propose a novel Social anchor-Unit GrAph Regularized Tensor Completion (SUGAR-TC) method to efficiently refine the tags of social images, which is insensitive to the scale of data. First, we construct an anchor-unit graph across multiple domains (e.g., image and user domains) rather than traditional anchor graph in a single domain. Second, a tensor completion based on Social anchor-Unit GrAph Regularization (SUGAR) is implemented to refine the tags of the anchor images. Finally, we efficiently assign tags to non-anchor images by leveraging the relationship between the non-anchor units and the anchor units. Experimental results on a real-world social image database well demonstrate the effectiveness and efficiency of SUGAR-TC, outperforming the state-of-the-art methods.