Mid-level Representation Research Articles

A mid-level video representation based on binary descriptors: A case study for pornography detection

With the growing amount of inappropriate content on the Internet, such as pornography, arises the need to detect and filter such material. The reason for this is given by the fact that such content is often prohibited in certain environments (e.g., schools and workplaces) or for certain publics (e.g., children). In recent years, many works have been mainly focused on detecting pornographic images and videos based on visual content, particularly on the detection of skin color. Although these approaches provide good results, they generally have the disadvantage of a high false positive rate since not all images with large areas of skin exposure are necessarily pornographic images, such as people wearing swimsuits or images related to sports. Local feature based approaches with Bag-of-Words models (BoW) have been successfully applied to visual recognition tasks in the context of pornography detection. Even though existing methods provide promising results, they use local feature descriptors that require a high computational processing time yielding high-dimensional vectors. In this work, we propose an approach for pornography detection based on local binary feature extraction and BossaNova image representation, a BoW model extension that preserves more richly the visual information. Moreover, we propose two approaches for video description based on the combination of mid-level representations namely BossaNova Video Descriptor (BNVD) and BoW Video Descriptor (BoW-VD). The proposed techniques are promising, achieving an accuracy of 92.40%, thus reducing the classification error by 16% over the current state-of-the-art local features approach on the Pornography dataset.

A comprehensive study on mid-level representation and ensemble learning for emotional analysis of video material

In today's society where audio-visual content such as professionally edited and user-generated videos is ubiquitous, automatic analysis of this content is a decisive functionality. Within this context, there is an extensive ongoing research about understanding the semantics (i.e., facts) such as objects or events in videos. However, little research has been devoted to understanding the emotional content of the videos. In this paper, we address this issue and introduce a system that performs emotional content analysis of professionally edited and user-generated videos. We concentrate both on the representation and modeling aspects. Videos are represented using mid-level audio-visual features. More specifically, audio and static visual representations are automatically learned from raw data using convolutional neural networks (CNNs). In addition, dense trajectory based motion and SentiBank domain-specific features are incorporated. By means of ensemble learning and fusion mechanisms, videos are classified into one of predefined emotion categories. Results obtained on the VideoEmotion dataset and a subset of the DEAP dataset show that (1) higher level representations perform better than low-level features, (2) among audio features, mid-level learned representations perform better than mid-level handcrafted ones, (3) incorporating motion and domain-specific information leads to a notable performance gain, and (4) ensemble learning is superior to multi-class support vector machines (SVMs) for video affective content analysis.

Pornographic image detection utilizing deep convolutional neural networks

Many internet users are potential victims of the pornographic images and a large part of them are underage children. Thus, content-based pornographic images detection is an important task in computer vision and multimedia research. Previous solutions usually rely on hand-engineered visual features that are hard to analyze and select. In this paper, to detect pornographic images in any style accurately and efficiently with a single model, a novel scheme utilizing the deep convolutional neural networks (CNN) is proposed. The training data are obtained from internet followed by an improved sliding window method and some novel data augmentation approaches. Then a highly efficient training algorithm is proposed based on two strategies. The first is the pre-trained mid-level representations non-fixed fine-tuning strategy. The second is adjusting the training data at the appropriate time on the basis of the performance of the proposed network on the validation set. Furthermore, we introduce a fast image scanning method which is also based on the sliding window approach in the test. We further propose a fast forward pass method based on the “fixed-point algorithm”. So our CNN could detect all scale images so fast by one forward pass. The effectiveness of the proposed method is demonstrated in experiments on the proposed dataset and the comparative results show that our method lead to state-of-the-art detection performance.

Semantic Classification of High-Resolution Remote-Sensing Images Based on Mid-level Features

With the resolution improvement of the remote-sensing images, more details are shown clearly. The challenge that comes along is how to boost the relatively low classification accuracy caused by using pixel-based image classification approaches and low-level visual structure. The low-level features (LLF) may not well describe the image due to the semantic gap between low-level visual features and high-level semantics of images. The bag-of-visual-words (BOV) model which generates mid-level features was proposed to bridge the two levels. However, it generally neglects the context information between local patches. In this paper, an object-oriented semantic classification algorithm that combines BOV with the optimal segmentation scale is presented. In this algorithm, BOV addresses the problem of the representation of mid-level for scenes, while the optimal segmentation scale intends to overcome the defect of conventional BOV in lacking of relationship between image patches and to give more thorough description. The object-based BOV is presented to construct mid-level representations for object description instead of LLF, and histogram intersection kernel (HIK) is introduced in support vector machine (SVM) for classification. The experiments conducted on three datasets testify the superiority of the proposed algorithm.

Strokelets: A Learned Multi-Scale Mid-Level Representation for Scene Text Recognition.

In this paper, we are concerned with the problem of automatic scene text recognition, which involves localizing and reading characters in natural images. We investigate this problem from the perspective of representation and propose a novel multi-scale representation, which leads to accurate, robust character identification and recognition. This representation consists of a set of mid-level primitives, termed strokelets, which capture the underlying substructures of characters at different granularities. The Strokelets possess four distinctive advantages: 1) usability: automatically learned from character level annotations; 2) robustness: insensitive to interference factors; 3) generality: applicable to variant languages; and 4) expressivity: effective at describing characters. Extensive experiments on standard benchmarks verify the advantages of the strokelets and demonstrate the effectiveness of the text recognition algorithm built upon the strokelets. Moreover, we show the method to incorporate the strokelets to improve the performance of scene text detection.

A New Angle on Object-Background Effects in Vection.

We considered whether optic flow generated by 3D relief of a foreground surface might influence visually-mediated self-motion perception (vection). We generated background motion consistent with self-rotation, and a foreground object with bumpy relief was either rotated with the observer (ego-centric) or fixed in world coordinates (world-centric). We found that vection strength ratings were greater in conditions with world-centric retinal motion of the foreground object, despite generating flow that was opposite to background motion. This effect was explained by observer judgments of the axis self-rotation in depth; whereas ego-centric flow generated experiences of more on-axis self-rotation, world-centric flow generated experiences of centrifugal rotation around the foreground object. These data suggest that foreground object motion can increase the perception of self-motion generated by optic flow, even when they reduce net retinal motion coherence and promote conditions for multisensory conflict. This finding supports the view that self-motion perception depends on mid-level representations of whole-scene motion.

Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation

Convolutional neural network (CNN) has achieved state-of-the-art performance in many different visual tasks. Learned from a large-scale training dataset, CNN features are much more discriminative and accurate than the hand-crafted features. Moreover, CNN features are also transferable among different domains. On the other hand, traditional dictionarybased features (such as BoW and SPM) contain much more local discriminative and structural information, which is implicitly embedded in the images. To further improve the performance, in this paper, we propose to combine CNN with dictionarybased models for scene recognition and visual domain adaptation. Specifically, based on the well-tuned CNN models (e.g., AlexNet and VGG Net), two dictionary-based representations are further constructed, namely mid-level local representation (MLR) and convolutional Fisher vector representation (CFV). In MLR, an efficient two-stage clustering method, i.e., weighted spatial and feature space spectral clustering on the parts of a single image followed by clustering all representative parts of all images, is used to generate a class-mixture or a classspecific part dictionary. After that, the part dictionary is used to operate with the multi-scale image inputs for generating midlevel representation. In CFV, a multi-scale and scale-proportional GMM training strategy is utilized to generate Fisher vectors based on the last convolutional layer of CNN. By integrating the complementary information of MLR, CFV and the CNN features of the fully connected layer, the state-of-the-art performance can be achieved on scene recognition and domain adaptation problems. An interested finding is that our proposed hybrid representation (from VGG net trained on ImageNet) is also complementary with GoogLeNet and/or VGG-11 (trained on Place205) greatly.

Category co-occurrence modeling for large scale scene recognition

Scene recognition involves complex reasoning from low-level local features to high-level scene categories. The large semantic gap motivates that most methods model scenes resorting to mid-level representations (e.g. objects, topics). However, this implies an additional mid-level vocabulary and has implications in training and inference. In contrast, the semantic multinomial (SMN) represents patches directly in the scene-level semantic space, which leads to ambiguity when aggregated to a global image representation. Fortunately, this ambiguity appears in the form of scene category co-occurrences which can be modeled a posteriori with a classifier. In this paper we observe that these patterns are essentially local rather than global, sparse, and consistent across SMNs obtained from multiple visual features. We propose a co-occurrence modeling framework where we exploit all these patterns jointly in a common semantic space, combining both supervised and unsupervised learning. Based on this framework we can integrate multiple features and design embeddings for large scale recognition directly in the scene-level space. Finally, we use the co-occurrence modeling framework to develop new scene representations, which experiments show that outperform previous SMN-based representations.

In-field automatic observation of wheat heading stage using computer vision

Growth stage information is an important factor for precision agriculture. It provides accurate evidence for agricultural management as well as early evaluation of yield. However, the observation of critical growth stages mainly relies on manual labour at present. This has some limitations because it is time-consuming, discontinuous and non-objective. Computer vision technology can help to alleviate these difficulties when monitoring growth status. This paper describes a novel automatic observation system for wheat heading stage based on computer vision. Images compliant with statistical requirements are taken in natural conditions where illumination changes frequently. Wheat plants with low spatial resolution overlap substantially, which increases observational difficulties. To adapt to the complex environment, a two-step coarse-to-fine wheat ear detection mechanism is proposed. In the coarse-detection step, machine learning technology is used to emphasise the candidate ear regions. In the fine-detection step, non-ear areas are eliminated through higher-level features. For that purpose, scale-invariant feature transform (SIFT) is densely extracted as the low-level visual descriptor, then Fisher vector (FV) encoding is employed to generate the mid-level representation. Based on three consecutive year's data of seven image sequences, a series of experiments are conducted to demonstrate the effectiveness and robustness of our proposition. Experimental results show that the proposed method significantly outperforms other existing methods with an average value of absolute error of 1.14 days on the test dataset. The results indicate that automatic observation is quite acceptable compared to manual observations.

Human action recognition with graph-based multiple-instance learning

A new approach to human action recognition from realistic videos is presented in this paper. First, an affine motion model is utilized to compensate background motion for the purpose of extracting dense foreground trajectories. Then, a trajectory spectral embedding is introduced to split up foreground action into multiple spatio-temporal action parts for constructing a mid-level representation. To deal with over-segmentation, a novel density discontinuity detector is proposed for the sake of generating semantically salient action parts. Finally, to handle the ambiguity in the training set, action classification is formulated within the multiple-instance learning framework, which a spatio-temporal graph model is incorporated into. Extensive experiments show that the proposed approach achieves competitive results to state of the art on UCF Sports, Kisses/Slaps, YouTube, and Hollywood datasets.

Script identification in the wild via discriminative convolutional neural network

Script identification facilitates many important applications in document/video analysis. This paper investigates a relatively new problem: identifying scripts in natural images. The basic idea is combining deep features and mid-level representations into a globally trainable deep model. Specifically, a set of deep feature maps is firstly extracted by a pre-trained CNN model from the input images, where the local deep features are densely collected. Then, discriminative clustering is performed to learn a set of discriminative patterns based on such local features. A mid-level representation is obtained by encoding the local features based on the learned discriminative patterns (codebook). Finally, the mid-level representations and the deep features are jointly optimized in a deep network. Benefiting from such a fine-grained classification strategy, the optimized deep model, termed Discriminative Convolutional Neural Network (DisCNN), is capable of effectively revealing the subtle differences among the scripts difficult to be distinguished, e.g. Chinese and Japanese. In addition, a large scale dataset containing 16,291 in-the-wild text images in 13 scripts, namely SIW-13, is created for evaluation. Our method is not limited to identifying text images, and performs effectively on video and document scripts as well, not requiring any preprocess like binarization, segmentation or hand-crafted features. The experimental comparisons on the datasets including SIW-13, CVSI-2015 and Multi-Script consistently demonstrate DisCNN a state-of-the-art approach for script identification.

Building Hierarchical Representations for Oracle Character and Sketch Recognition.

In this paper, we study oracle character recognition and general sketch recognition. First, a data set of oracle characters, which are the oldest hieroglyphs in China yet remain a part of modern Chinese characters, is collected for analysis. Second, typical visual representations in shape- and sketch-related works are evaluated. We analyze the problems suffered when addressing these representations and determine several representation design criteria. Based on the analysis, we propose a novel hierarchical representation that combines a Gabor-related low-level representation and a sparse-encoder-related mid-level representation. Extensive experiments show the effectiveness of the proposed representation in both oracle character recognition and general sketch recognition. The proposed representation is also complementary to convolutional neural network (CNN)-based models. We introduce a solution to combine the proposed representation with CNN-based models, and achieve better performances over both approaches. This solution has beaten humans at recognizing general sketches.

Learning shared, discriminative, and compact representations for visual recognition.

Dictionary-based and part-based methods are among the most popular approaches to visual recognition. In both methods, a mid-level representation is built on top of low-level image descriptors and high-level classifiers are trained on top of the mid-level representation. While earlier methods built the mid-level representation without supervision, there is currently great interest in learning both representations jointly to make the mid-level representation more discriminative. In this work we propose a new approach to visual recognition that jointly learns a shared, discriminative, and compact mid-level representation and a compact high-level representation. By using a structured output learning framework, our approach directly handles the multiclass case at both levels of abstraction. Moreover, by using a group-sparse prior in the structured output learning framework, our approach encourages sharing of visual words and thus reduces the number of words used to represent each class. We test our proposed method on several popular benchmarks. Our results show that, by jointly learning mid- and high-level representations, and fostering the sharing of discriminative visual words among target classes, we are able to achieve state-of-the-art recognition performance using far less visual words than previous approaches.

Partial-Duplicate Clustering and Visual Pattern Discovery on Web Scale Image Database

In this paper, we study the problem of discovering visual patterns and partial-duplicate images, which is fundamental to visual concept representation and image parsing, but very challenging when the database is extremely large, such as billions of images indexed by a commercial search engine. Although extensive research with sophisticated algorithms has been conducted for either partial-duplicate clustering or visual pattern discovery, most of them can not be easily extended to this scale, since both are clustering problems in nature and require pairwise comparisons. To tackle this computational challenge, we introduce a novel and highly parallelizable framework to discover partial-duplicate images and visual patterns in a unified way in distributed computing systems. We emphasize the nested property of local features, and propose the generalized nested feature (GNF) as a mid-level representation for regions and local patterns. Initial coarse clusters are then discovered by GNFs, upon which $n$ -gram GNF is defined to represent co-occurrent visual patterns. After that, efficient merging and refining algorithms are used to get the partial-duplicate clusters, and logical combinations of probabilistic GNF models are leveraged to represent the visual patterns of partially duplicate images. Extensive experiments show the parallelizable property and effectiveness of the algorithms on both partial-duplicate clustering and visual pattern discovery. With 2000 machines, it costs about eight and 400 minutes to process one million and 40 million images respectively, which is quite efficient compared to previous methods.

Understanding mid-level representations in visual processing.

It is clear that early visual processing provides an image-based representation of the visual scene: Neurons in Striate cortex (V1) encode nothing about the meaning of a scene, but they do provide a great deal of information about the image features within it. The mechanisms of these "low-level" visual processes are relatively well understood. We can construct plausible models for how neurons, up to and including those in V1, build their representations from preceding inputs down to the level of photoreceptors. It is also clear that at some point we have a semantic, "high-level" representation of the visual scene because we can describe verbally the objects that we are viewing and their meaning to us. A huge number of studies are examining these "high-level" visual processes each year. Less well studied are the processes of "mid-level" vision, which presumably provide the bridge between these "low-level" representations of edges, colors, and lights and the "high-level" semantic representations of objects, faces, and scenes. This article and the special issue of papers in which it is published consider the nature of "mid-level" visual processing and some of the reasons why we might not have made as much progress in this domain as we would like.

Building Effective Representations for Sketch Recognition

As the popularity of touch-screen devices, understanding a user's hand-drawn sketch has become an increasingly important research topic in artificial intelligence and computer vision. However, different from natural images, the hand-drawn sketches are often highly abstract, with sparse visual information and large intra-class variance, making the problem more challenging. In this work, we study how to build effective representations for sketch recognition. First, to capture saliency patterns of different scales and spatial arrangements, a Gabor-based low-level representation is proposed. Then, based on this representation, to discovery more complex patterns in a sketch, a Hybrid Multilayer Sparse Coding (HMSC) model is proposed to learn mid-level representations. An improved dictionary learning algorithm is also leveraged in HMSC to reduce overfitting to common but trivial patterns. Extensive experiments show that the proposed representations are highly discriminative and lead to large improvements over the state of the arts.

Towards robust automatic affective classification of images using facial expressions for practical applications

Affect is an important feature of multimedia content and conveys valuable information for multimedia indexing and retrieval. Most existing studies for affective content analysis are limited to low-level features or mid-level representations, and are generally criticized for their incapacity to address the gap between low-level features and high-level human affective perception. The facial expressions of subjects in images carry important semantic information that can substantially influence human affective perception, but have been seldom investigated for affective classification of facial images towards practical applications. This paper presents an automatic image emotion detector (IED) for affective classification of practical (or non-laboratory) data using facial expressions, where a lot of real-world challenges are present, including pose, illumination, and size variations etc. The proposed method is novel, with its framework designed specifically to overcome these challenges using multi-view versions of face and fiducial point detectors, and a combination of point-based texture and geometry. Performance comparisons of several key parameters of relevant algorithms are conducted to explore the optimum parameters for high accuracy and fast computation speed. A comprehensive set of experiments with existing and new datasets, shows that the method is effective despite pose variations, fast, and appropriate for large-scale data, and as accurate as the method with state-of-the-art performance on laboratory-based data. The proposed method was also applied to affective classification of images from the British Broadcast Corporation (BBC) in a task typical for a practical application providing some valuable insights.

Multi-level max-margin analysis for semantic classification of satellite images

The performance of scene classification of satellite images strongly relies on the discriminative power of the low-level and mid-level feature representation. This paper presents a novel approach, named multi-level max-margin analysis (M3DA) for semantic classification for high-resolution satellite images. In our M3DA model, the maximum entropy discrimination latent Dirichlet allocation (MedLDA) model is applied to learn the topic-level features first, and then based on a bag-of-words representation of low-level local image features, the large margin nearest neighbor (LMNN) classifier is used to optimize a multiple soft label composed of word-level features (generated by SVM classifier) and topic-level features. The categorization performances on 21-class land-use dataset have demonstrated that the proposed model in multi-level max-margin scheme can distinguish different categories of land-use scenes reasonably.

Research on Summary Highlight Ranking of Sports Video

A novel multimodal approach of highlight ranking for sports video summaries in affective context was proposed based on player behavior information and audio keywords of sports game. The mid-level representation trajectory-action-audio is constructed for the video content by fusing the information of player trajectory, action and audio keywords. Based on trajectory-action-audio, the computational affective features are extracted to describe the objective process of highlight ranking of sports video summaries from user subjective perception. A kernel based nonlinear probabilistic ranking model construction method is proposed, which is robust for the noisy data and provided with good expansibility. In addition, a new subjective evaluation criterion is proposed to guide model construction and feature extraction with the assistance of forward search algorithm.

Towards a model for mid-level feature representation of scenes

Towards a model for mid-level feature representation of scenes