Web Image Research Articles

Affective image classification via semi-supervised learning from web images

Affective image classification has drawn increasing research attentions in the affective computing and multimedia communities. Despite many solutions proposed in the literature, it remains a major challenge to bridge the semantic gap between visual features of images and their affective characteristics, partly due to the lack of adequate training samples, which can be largely ascribed to the all-consuming nature of affective image annotation. In this paper, we propose a novel affective image classification algorithm based on semi-supervised learning from web images (SSL-WI). This algorithm consists of four major steps, including color and texture feature extraction, baseline classifier construction, feature selection, and jointly using training images and retrieved web images to re-train the classifier. We have applied this algorithm, the baseline classifier that is not trained by web images, and two state-of-the-art algorithms to differentiating color images in a three-dimensional discrete emotional space. Our results suggest that, with the scheme of semi-supervised learning from web images, the proposed algorithm is able to produce more accurate affective image classification than other three approaches.

What can we learn about the female condom online? An analysis of visual representations of the female condom on the Internet

ABSTRACTThis study used social functional semiotics and the concept of gender display to examine visual representations of the female condom on Google Images. A content analysis of 195 web images identified that almost half of the images were about the appearance theme of the female condom followed by the how-to-use theme. A majority of visuals did not have captions. Humans were present in 64.1% of the images, with a majority of visuals depicting dismembered female body parts rather than a whole person. These and other findings are discussed in regards to theoretical and health message design implications.

Deepdiary: Lifelogging image captioning and summarization

Automatic image captioning has been studied extensively over the last few years, driven by breakthroughs in deep learning-based image-to-text translation models. However, most of this work has considered captioning web images from standard data sets like MS-COCO, and has considered single images in isolation. To what extent can automatic captioning models learn finer-grained contextual information specific to a given person’s day-to-day visual experiences? In this paper, we consider captioning image sequences collected from wearable, life-logging cameras. Automatically-generated captions could help people find and recall photos among their large-scale life-logging photo collections, or even to produce textual “diaries” that summarize their day. But unlike web images, photos from wearable cameras are often blurry and poorly composed, without an obvious single subject. Their content also tends to be highly dependent on the context and characteristics of the particular camera wearer. To address these challenges, we introduce a technique to jointly caption sequences of photos, which allows captions to take advantage of temporal constraints and evidence across time, and we introduce a technique to increase the diversity of generated captions, so that they can describe a photo from multiple perspectives (e.g., first-person versus third-person). To test these techniques, we collect a dataset of about 8000 realistic lifelogging images, a subset of which are annotated with nearly 5000 human-generated reference sentences. We evaluate the quality of image captions both quantitatively and qualitatively using Amazon Mechanical Turk, finding that while these algorithms are not perfect, they could be an important step towards helping to organize and summarize lifelogging photos.

Large-scale k-means clustering via variance reduction

Abstract With the increase of the volume of data such as images in web, it is challenging to perform k-means clustering on millions or even billions of images efficiently. One of the reasons is that k-means requires to use a batch of training data to update cluster centers at every iteration, which is time-consuming. Conventionally, k-means is accelerated by using one or a mini-batch of instances to update the centers, which leads to a bad performance due to the stochastic noise. In the paper, we decrease such stochastic noise, and accelerate k-means by using variance reduction technique. Specifically, we propose a position correction mechanism to correct the drift of the cluster centers, and propose a variance reduced k-means named VRKM. Furthermore, we optimize VRKM by reducing its computational cost, and propose a new variant of the variance reduced k-means named VRKM++. Comparing with VRKM, VRKM++ does not have to compute the batch gradient, and is more efficient. Extensive empirical studies show that our methods VRKM and VRKM++ outperform the state-of-the-art method, and obtain about 2 × and 4 × speedups for large-scale clustering, respectively. The source code is available at https://www.github.com/YaweiZhao/VRKM_sofia-ml .

Extracting Key Segments of Videos for Event Detection by Learning From Web Sources

In this paper, we present a novel approach of extracting the key segments for event detection in unconstrained videos. The key segments are automatically extracted by transferring the knowledge learned from Web images and Web videos to consumer videos. We propose an adaptive latent structural support vector machine model, where the locations of key segments in videos are regarded as latent variables due to the unavailability of the ground truth of key-segment locations in training data. In order to alleviate the time-consuming and labor-expensive manual annotation of huge amounts of training videos, a large number of loosely labeled Web images as well as videos are collected from the Web sources. Additionally, a limited number of labeled consumer videos are utilized to guarantee the precision of the model. Considering the semantic diversity of key segments, we learn a set of concepts as the semantic description of key segments and explore the temporal information of concepts to capture the sequential relations between the segments. The concepts are automatically discovered by using Web images and videos with their associated tags and description sentences. Comprehensive experiments on the Columbia's consumer video and the TRECVID 2014 Multimedia Event Detection datasets demonstrate that our method outperforms the state-of-the-art methods.

STUDY OF SOCIAL IMAGE RE-RANKING ACCORDING INTER AND INTRA USER IMPACT

Social media provides most of the websites for example Facebook that allows users to describe their images with special tag. That helps in the web image retrival system tag based image search is very important method to find out images uploaded by the social users in most of social websites that helps to make the result related and improve the variety of the images. we considered the visual information and click count of the images in this system first we sort the images by inter-user ranking the user who upload the maximum image will rank higher. After this we implement intra-user ranking on the ranked user image set and only the most related image from each user image database is selected . These selected images combine the final retrived results. We build an inverted index for social image databases to improve the speed of searching of an images.

Discovering and Distinguishing Multiple Visual Senses for Polysemous Words

To reduce the dependence on labeled data, there have been increasing research efforts on learning visual classifiers by exploiting web images. One issue that limits their performance is the problem of polysemy. To solve this problem, in this work, we present a novel framework that solves the problem of polysemy by allowing sense-specific diversity in search results. Specifically, we first discover a list of possible semantic senses to retrieve sense-specific images. Then we merge visual similar semantic senses and prune noises by using the retrieved images. Finally, we train a visual classifier for each selected semantic sense and use the learned sense-specific classifiers to distinguish multiple visual senses. Extensive experiments on classifying images into sense-specific categories and re-ranking search results demonstrate the superiority of our proposed approach.

Web image re-ranking using query specific in cloud computing

Question answering (QA) allows all users to get information in enhanced technique. In this project we suggest a system for inspiring textual answer with appropriate media data. Our system consists of three components Interpretation median picking, Inquiry propagation, Data pick and Launching. Interpretation median picking is used to select various types of answers. Inquiry propagation is used for extracting the root words from the given query. Data pick and Launching is used for selecting the appropriate answer and producing the result. We use Stemming algorithm, Naïve Bayes classifier algorithm and page ranking algorithms. Stemming algorithm is used to extract the root word from the given searched query. Naïve Bayes classifier algorithm is used for selecting the type of medium. By using the page ranking algorithm the optimal solution is got. Our approach automatically determines which media will be a best solution for the given query. It automatically harvests the data from website for getting the answer. Our approach can enable a novel multimedia question answering (MMQA) approach as users can find multimedia answers by matching their questions with those in the pool. We are enhancing community contributed answers. Any user who is unaware of data can get the information promptly. Our approach is to deal with the complex questions in an effective way. Based on the generated queries, we vertically collect image and video data with multimedia search engines.

Creating personalized video summaries via semantic event detection

Video summarization has great potential in many application areas that enable fast browsing and efficient video indexing. Viewers prefer to browse a video summary containing the contents that they enjoy since watching an entire video may be time-consuming. We believe that it is necessary to create an automated tool that is capable of generating personalized video summaries. In this paper, we propose a new event detection-based personalized video summarization framework and deploy it to create film and soccer video summaries. In order to obtain effective event detection performance, we introduce two transfer learning method. The first event detection method is achieved based on the combination of convolutional neural network and support vector machine (CNNs–SVM). The second method is achieved using a fine-tuned summarization network (SumNet) that fuses fine-tuned object and scene networks. In this study, the training data consists of two datasets: (1) a 21K set of web images of back hugging, hand shaking, and standing talking used to detect a film event, and (2) a 30K set of web soccer match images of goals, fouls, and yellow cards to detect soccer events. Given an original video, we first segment it into shots and then use the trained model for event detection. Finally, based on the specification of user preferences, we generate a personalized event-based summary. We test our framework with several film videos and soccer videos. Experimental results demonstrate that the proposed fine-tuned SumNet achieves the best performance of 96.88% and $$98.50\%$$ , which is effective for generating personalized video summaries.

Hypergraph dominant set based multi-video summarization

Multi-Video Summarization (MVS) aims at condensing a large number of web videos by the same search query into a compact storyboard or video skimming. Although video summarization has achieved great progress, most existing work focuses only on extracting a summary from a single video. The genre-specific assumptions for Single Video Summarization (SVS) often do not hold for MVS. Moreover, the complex data characteristics among web videos make MVS more challenging. To well address the challenges in MVS, this paper presents a novel framework with Hypergraph Dominant Set (HDS), termed MVS-HDS. Specifically, we first formulate the MVS task into a problem of finding dominant sets in a hypergraph. Then, with the help of web images searched by the same query, a Query Dependent Maximum Marginal Relevance (QD-MMR) method is developed to refine these keyframe candidates by balancing the criteria of conciseness and query adaptation. Finally, to make the summary easy to understand, we further present a Graph-based Topical Closeness (GTC) method to make it more comprehensible. Extensive experiments show that the proposed framework clearly outperforms the state-of-the-art methods.

A recursive framework for expression recognition: from web images to deep models to game dataset

In this paper, we propose a recursive framework to recognize facial expressions from images in real scenes. Unlike traditional approaches that typically focus on developing and refining algorithms for improving recognition performance on an existing dataset, we integrate three important components in a recursive manner: facial dataset generation, facial expression recognition model building, and interactive interfaces for testing and new data collection. To start with, we first create candid images for facial expression (CIFE) dataset. We then apply a convolutional neural network (CNN) to CIFE and build a CNN model for web image expression classification. In order to increase the expression recognition accuracy, we also fine-tune the CNN model and thus obtain a better CNN facial expression recognition model. Based on the fine-tuned CNN model, we design a facial expression game engine and collect a new and more balanced dataset, GaMo. The images of this dataset are collected from the different expressions our game users make when playing the game. Finally, we run yet another recursive step—a self-evaluation of the quality of the data labeling and propose a self-cleansing mechanism for improve the quality of the data. We evaluate the GaMo and CIFE datasets and show that our recursive framework can help build a better facial expression model for dealing with real scene facial expression tasks.

CISRDCNN: Super-resolution of compressed images using deep convolutional neural networks

In recent years, much research has been conducted on image super-resolution (SR). To the best of our knowledge, however, few SR methods were concerned with compressed images. The SR of compressed images is a challenging task due to the complicated compression artifacts, while many images suffer from them in practice. The intuitive solution for this difficult task is to decouple it into two sequential but independent subproblems, i.e., compression artifacts reduction (CAR) and SR. Nevertheless, some useful details may be removed in CAR stage, which is contrary to the goal of SR and makes the SR stage more challenging. In this paper, an end-to-end trainable deep convolutional neural network is designed to perform SR on compressed images (CISRDCNN), which reduces compression artifacts and improves image resolution jointly. Experiments on compressed images produced by JPEG (we take the JPEG as an example in this paper) demonstrate that the proposed CISRDCNN yields state-of-the-art SR performance on commonly used test images and imagesets. The results of CISRDCNN on real low quality web images are also very impressive, with obvious quality enhancement. Further, we explore the application of the proposed SR method in low bit-rate image coding, leading to better rate-distortion performance than JPEG.

Knowledge-Based Topic Model for Unsupervised Object Discovery and Localization.

Unsupervised object discovery and localization is to discover some dominant object classes and localize all of object instances from a given image collection without any supervision. Previous work has attempted to tackle this problem with vanilla topic models, such as latent Dirichlet allocation (LDA). However, in those methods no prior knowledge for the given image collection is exploited to facilitate object discovery. On the other hand, the topic models used in those methods suffer from the topic coherence issue-some inferred topics do not have clear meaning, which limits the final performance of object discovery. In this paper, prior knowledge in terms of the so-called must-links are exploited from Web images on the Internet. Furthermore, a novel knowledge-based topic model, called LDA with mixture of Dirichlet trees, is proposed to incorporate the must-links into topic modeling for object discovery. In particular, to better deal with the polysemy phenomenon of visual words, the must-link is re-defined as that one must-link only constrains one or some topic(s) instead of all topics, which leads to significantly improved topic coherence. Moreover, the must-links are built and grouped with respect to specific object classes, thus the must-links in our approach are semantic-specific, which allows to more efficiently exploit discriminative prior knowledge from Web images. Extensive experiments validated the efficiency of our proposed approach on several data sets. It is shown that our method significantly improves topic coherence and outperforms the unsupervised methods for object discovery and localization. In addition, compared with discriminative methods, the naturally existing object classes in the given image collection can be subtly discovered, which makes our approach well suited for realistic applications of unsupervised object discovery.Unsupervised object discovery and localization is to discover some dominant object classes and localize all of object instances from a given image collection without any supervision. Previous work has attempted to tackle this problem with vanilla topic models, such as latent Dirichlet allocation (LDA). However, in those methods no prior knowledge for the given image collection is exploited to facilitate object discovery. On the other hand, the topic models used in those methods suffer from the topic coherence issue-some inferred topics do not have clear meaning, which limits the final performance of object discovery. In this paper, prior knowledge in terms of the so-called must-links are exploited from Web images on the Internet. Furthermore, a novel knowledge-based topic model, called LDA with mixture of Dirichlet trees, is proposed to incorporate the must-links into topic modeling for object discovery. In particular, to better deal with the polysemy phenomenon of visual words, the must-link is re-defined as that one must-link only constrains one or some topic(s) instead of all topics, which leads to significantly improved topic coherence. Moreover, the must-links are built and grouped with respect to specific object classes, thus the must-links in our approach are semantic-specific, which allows to more efficiently exploit discriminative prior knowledge from Web images. Extensive experiments validated the efficiency of our proposed approach on several data sets. It is shown that our method significantly improves topic coherence and outperforms the unsupervised methods for object discovery and localization. In addition, compared with discriminative methods, the naturally existing object classes in the given image collection can be subtly discovered, which makes our approach well suited for realistic applications of unsupervised object discovery.

Decision Support for Grape Crop Protection Using Ontology

Weather based decision support for managing pests and diseases of crops requires use of information technology. This paper details a system developed using ontology, semantic web rule language and image processing techniques for management of pests and diseases on wines, particularly in hot tropical region of India. It aims at minimising use of pesticides and fungicides by forecasting pests and diseases occurrence using information about meteorological conditions and it's relation with pest and disease occurrence. It is named as PDMGrapes. For system knowledge base, knowledge available in different formats on grape pests and diseases is converted to ontology. Favourable meteorological conditions for pest and disease occurrences are mentioned by SWRL rules. Grapes disease identification is done using image processing techniques. The system helps grape growers to minimise side effects of pesticides on environment. The developed system is validated and verified for accuracy and performance.

A Benchmark Dataset and Learning High-Level Semantic Embeddings of Multimedia for Cross-Media Retrieval

The selection of semantic concepts for modal construction and data collection remains an open research issue. It is highly demanding to choose good multimedia concepts with small semantic gaps to facilitate the work of cross-media system developers. However, very little work has been done in this area. This paper contributes a new, real-world web image dataset for cross-media retrieval called FB5K. The proposed FB5K dataset contains the following attributes: 1) 5130 images crawled from Facebook; 2) images that are categorized according to users’ feelings; 3) images independent of text and language rather than using feelings for search. Furthermore, we propose a novel approach through the use of Optical Character Recognition and explicit incorporation of high-level semantic information. We comprehensively compute the performance of four different subspace-learning methods and three modified versions of the Correspondence Auto Encoder, alongside numerous text features and similarity measurements comparing Wikipedia, Flickr30k, and FB5K. To check the characteristics of FB5K, we propose a semantic-based cross-media retrieval method. To accomplish cross-media retrieval, we introduced a new similarity measurement in the embedded space, which significantly improved system performance compared with the conventional Euclidean distance. Our experimental results demonstrated the efficiency of the proposed retrieval method on three different datasets to simplify and improve general image retrieval.

Image-Matching Based Identification of Store Signage Using Web-Crawled Information

We address automatic matching of street images with relevant web resources to enable the identification of store signage in street images. Identification methods for signage usually involve image matching, which attempts to match query images to other similar viewings using pre-labeled copies from a target data set. Manual target data set, such as a fingerprinting database can ensure high-quality data but collected data must be fed manually, which significantly adds costs. Utilizing web-crawled information is a way for automatic data set generation at lower cost, however, imbalanced and noisy data can adversely affect identification accuracy. Our work aims to resolve these issues. We propose a signage identifier in Web-crawled information – SIWI. The SIWI includes a web image data set construction method, which can self-generate high-quality data sets through automated web-mining, including data filtering and pruning strategies, which effectively reduce the identification error caused by noise, imbalance, and insufficient data. Furthermore, by applying a Hybrid Image Matching method that combines the deep learning approach with the feature point matching to signage identification without Optical Character Recognition, it can handle arbitrary signage designs. Because there is no specialized training involved, the same process should also work for any other locations without manual adjustment. An experimental result achieves 91% accuracy in a real-life application, which confirms its effectiveness.

Clustering of near duplicate images using bundled features

Clustering the images is generally based on the image’s visual features. Selection of relevant features is the most essential task. A clustering approach based on the bundled features is presented in this paper. Bundling of affine scale invariant feature transform (ASIFT) feature helps to cluster the near duplicates. When the local features are combined with the ASIFT features, the clustering efficiency is increased. Clustering the results from the web image search engines is very essential to help users narrow their search. We applied our idea of clustering with bundled features over Google Image search results. The results obtained show that the presented approach outperforms compared to the clustering done only with local features.

Top-Down Neural Attention by Excitation Backprop

We aim to model the top-down attention of a Convolutional Neural Network (CNN) classifier for generating task-specific attention maps. Inspired by a top-down human visual attention model, we propose a new backpropagation scheme, called Excitation Backprop, to pass along top-down signals downwards in the network hierarchy via a probabilistic Winner-Take-All process. Furthermore, we introduce the concept of contrastive attention to make the top-down attention maps more discriminative. In experiments, we demonstrate the accuracy and generalizability of our method in weakly supervised localization tasks on the MS COCO, PASCAL VOC07 and ImageNet datasets. The usefulness of our method is further validated in the text-to-region association task. On the Flickr30k Entities dataset, we achieve promising performance in phrase localization by leveraging the top-down attention of a CNN model that has been trained on weakly labeled web images.

Accessible images (AIMS): a model to build self-describing images for assisting screen reader users

Non-visual web access depends on the textual description of various non-text elements of web pages. The existing methods of describing images for non-visual access do not provide a strong coupling between described images and their description. If an image is reused multiple times either in a single Web site or across multiple times, it is required to keep the description at all instances. This paper presents a tightly coupled model termed accessible images (AIMS) which utilizes a steganography-based approach to embed the description in the images at the server side and updating alt text of the web pages with the description extracted with the help of a browser extension. The proposed AIMS model has been built, targeting toward a web image description ecosystem in which images evolve into a self-description phase. The primary advantage of the proposed AIMS model is the elimination of the redundant description of an image resource at multiple instances. The experiments conducted on a dataset confirm that the AIMS model is capable of embedding and extracting descriptions with an accuracy level of 99.6%.

Taking advantage of multi-regions-based diagonal texture structure descriptor for image retrieval

With the rapid development of information technology, the capacity of Web images database becomes larger and larger. How to quickly and effectively find the desired images in Web image databases becomes the challenge needed to resolve with high priority. In this paper a novel diagonal texture structure descriptor (DTSD) is proposed, and a new framework considering hue, saturation and value components is utilized for image retrieval. In specific, we firstly use Otsu algorithm to segment image into foreground and background, and the features of multi-regions are respectively considered. That is, we present the contents of these multi-regions distinctively to reduce the influence of each other, which would perform hierarchical feature description and realize more accurate content match for image retrieval. In this study, to simulate the characteristic of human eyes for perceiving colors, hue and saturation components are quantized into various bins which can obtain more detailed description for color difference. Meanwhile, DTSD is extracted based on value component to represent the edge information as the feature of receptive field. Such a method can improve the spatial resolution ability of the descriptor, and identify finer structure of an image. Moreover, histogram with respect to these three components, i.e., hue, saturation and value, is utilized to generate the feature vector of an image. We carry out the experiments on benchmark Corel and UCID image datasets, and the extensive experimental results demonstrate that our method achieves better performance in comparison with state of the art image retrieval algorithms. The proposed method is very promising, which can provide more accurate retrieved results on the basis of color & texture descriptions in multi-regions, and further enhances the performance of the intelligent image retrieval system.