Agglomerative Information Bottleneck Research Articles

Identifying legitimate Web users and bots with different traffic profiles — an Information Bottleneck approach

Recent studies reported that about half of Web users nowadays are intelligent agents (Web bots). Many bots are impersonators operating at a very high sophistication level, trying to emulate navigational behaviors of legitimate users (humans). Moreover, bot technology continues to evolve which makes bot detection even harder. To deal with this problem, many advanced methods for differentiating bots from humans have been proposed, a large part of which relies on supervised machine learning techniques. In this paper, we propose a novel approach to identify various profiles of bots and humans which combines feature selection and unsupervised learning of HTTP-level traffic patterns to develop a user session classification model. Session clustering is performed with the agglomerative Information Bottleneck (aIB) algorithm, as well as with some other reference algorithms. The model is then used to classify new sessions to one of the profiles and to label the sessions as performed by bots or humans. An extensive experimental study, based on real server log data, demonstrates the ability of aIB clustering to distinguish user profiles and confirms high performance of the classification model in terms of accuracy, F1, recall, and precision.

Acoustic event diarization in TV/movie audios using deep embedding and integer linear programming

In this study, we propose a method for acoustic event diarization based on a feature of deep embedding and a clustering algorithm of integer linear programming. The deep embedding learned by deep auto-encoder network is used to represent the properties of different classes of acoustic events, and then the integer linear programming is adopted for merging audio segments belonging to the same class of acoustic events. Four kinds of TV/movie audios (21.5 h in total) are used as experimental data, including Sport, Situation comedy, Award ceremony, and Action movie. We compare the deep embedding with state-of-the-art features. Further, the clustering algorithm of integer linear programming is compared with other clustering algorithms adopted in previous works. Finally, the proposed method is compared to both supervised and unsupervised methods on four kinds of TV/movie audios. The results show that the proposed method is superior to other unsupervised methods based on agglomerative information bottleneck, Bayesian information criterion and spectral clustering, and is little inferior to the supervised method based on deep neural network in terms of acoustic event error.

The purpose driven privacy preservation for accelerometer-based activity recognition

Accelerometer-based activity recognition (AAR) attracted a lot of attentions due to the wide spread of smartphones with energy-efficiency. However, since accelerometer data contains individual characteristics; AAR might raise privacy concerns. Although numerous privacy preservation approaches, such as ”privacy filtering, differential privacy, and inferential privacy”, have been proposed to conceal sensitive information, unfortunately they cannot address the privacy problem associated with AAR. In this paper, we report our efforts to control the use of the AAR while preserving the privacy. To achieve this task, our method leverages a connection to agglomerative information bottleneck, through which the amount of disclosed data can be compressed so that irrelevant private information can be reduced, and a connection to general privacy statistical inference framework, where both of the privacy leakage and utility accuracy are considered as mutual information. Our experimental results have shown that the proposed solution can greatly reduce privacy leakage while maintaining a relative good utility.

Application of the Information Bottleneck method to discover user profiles in a Web store

ABSTRACTThe paper deals with the problem of discovering groups of Web users with similar behavioral patterns on an e-commerce site. We introduce a novel approach to the unsupervised classification of user sessions, based on session attributes related to the user click-stream behavior, to gain insight into characteristics of various user profiles. The approach uses the agglomerative Information Bottleneck (IB) algorithm. Based on log data for a real online store, efficiency of the approach in terms of its ability to differentiate between buying and non-buying sessions was validated, indicating some possible practical applications of the our method. Experiments performed for a number of session samples showed that the method is capable of separating both types of sessions to a large extent. A detailed analysis was performed for the number of clusters ranging from two to seven, and the results were compared to those achieved by applying the most common clustering algorithm, k-means. Increasing the number of clusters generally leads to better results for both algorithms. However, IB demonstrated much higher average efficiency than k-means for the corresponding number of clusters, and this superiority was especially clear for lower number of clusters. The IB-based division of user sessions into seven clusters gives the mean entropy value of 0.28, which means the 95% separation of sessions of both types. Furthermore, a big advantage of our approach is that it gives a possibility to analyze the probability distribution of session attributes in individual clusters, which allows one to discover hidden knowledge about common characteristics of various user profiles and use this knowledge to support managerial decisions.

Abnormal behavior detection in videos using deep learning

A new method for abnormal behavior detection is proposed using deep learning. Two SDAEs are utilized to automatically learn appearance feature and motion feature respectively, which are constrained in the space–time volume along dense trajectories that carry rich motion information to reduce the computational complexity. The vision words are exploited to describe behavior by the bag of words, and in order to reduce feature dimensions, the Agglomerative Information Bottleneck approach is used for vocabulary compression. An adaptive feature fusion method is adopted to enhance the discriminating power of these features. A sparse representation is exploited to abnormal behavior detection, which improve the detection accuracy. The proposed method is verified on the public dataset BEHAVE and BOSS and the results indicate the effectiveness of the proposed method.

Distributed clustering of categorical data using the information bottleneck framework

We perform clustering of categorical data using the Information Bottleneck, (IB), framework at large scale. We examine the performance of existing solutions using multiple machine architectures. The IB method uses information theory to recast database relations as probability distributions and the proximity of their tuples as their loss of information when they are considered together. More precisely, we study the Agglomerative Information Bottleneck, the Sequential Information Bottleneck and LIMBO, a newer approach that uses summaries of the original data. First we evaluate the performance and limitations of these algorithms when confronted with large datasets in a single, powerful machine. We then propose new implementations that take advantage of distributed environments. Using real and large synthetic datasets of tens of Gigabytes in size, we finally evaluate their effectiveness and efficiency.

Privacy-Guaranteed Two-Agent Interactions Using Information-Theoretic Mechanisms

This paper introduces a multi-round interaction problem with privacy constraints between two agents that observe correlated data. The data are assumed to have both public and private features, and the goal of the interaction is to share the public data subject to utility constraints (bounds on the distortion of public feature) while ensuring bounds on the information leakage of the private data at the other agent. The agents alternately share data with one another for a total of $K$ rounds such that each agent initiates sharing over $K/2$ rounds. The interactions are modeled as a collection of $K$ random mechanisms (mappings), one for each round. The goal is to jointly design the $K$ private mechanisms to determine the set of all achievable distortion-leakage pairs at each agent. Arguing that a mutual information-based leakage metric can be appropriate for streaming data settings, this paper: 1) determines the set of all achievable distortion-leakage tuples; 2) shows that the $K$ mechanisms allow for precisely composing the total privacy budget over $K$ rounds without loss; and 3) develops conditions under which interaction reduces the net leakage at both agents and illustrates it for a specific class of sources. The paper then focuses on log-loss distortion to better understand the effect on leakage of using a commonly used utility metric in learning theory. The resulting interaction problem leads to a non-convex sum-leakage-distortion optimization problem that can be viewed as an interactive version of the information bottleneck problem. A new merge-and-search algorithm that extends the classical agglomerative information bottleneck algorithm to the interactive setting is introduced to determine a provable locally optimal solution. Finally, the benefit of interaction under log-loss is illustrated for specific source classes, and the optimality of one-shot is proved for Gaussian sources under both mean-square and log-loss distortions constraints.

Brain parcellation based on information theory

Background and objectiveIn computational neuroimaging, brain parcellation methods subdivide the brain into individual regions that can be used to build a network to study its structure and function. Using anatomical or functional connectivity, hierarchical clustering methods aim to offer a meaningful parcellation of the brain at each level of granularity. However, some of these methods have been only applied to small regions and strongly depend on the similarity measure used to merge regions. The aim of this work is to present a robust whole-brain hierarchical parcellation that preserves the global structure of the network. MethodsBrain regions are modeled as a random walk on the connectome. From this model, a Markov process is derived, where the different nodes represent brain regions and in which the structure can be quantified. Functional or anatomical brain regions are clustered by using an agglomerative information bottleneck method that minimizes the overall loss of information of the structure by using mutual information as a similarity measure. ResultsThe method is tested with synthetic models, structural and functional human connectomes and is compared with the classic k-means. Results show that the parcellated networks preserve the main properties and are consistent across subjects. ConclusionThis work provides a new framework to study the human connectome using functional or anatomical connectivity at different levels.

Scene categorization based on compact SPM and ensemble of extreme learning machines

The performance of the typical scene categorization approach based on spatial pyramid model (SPM) and support vector machine (SVM) is limited in high-dimensional image representations and kernelized classifiers. In this article, a novel method combined with compact spatial pyramid model (CSPM) and ensemble of extreme learning machines (ELM) is proposed. The method consists of two major steps: First, Agglomerative Information Bottleneck (AIB) algorithm is applied to construct a compact spatial pyramid model (CSPM), which can compress the vocabulary size that maintains the performance of original SPM. Second, an effective ensemble method combined with Rotation Forest and weighted voting scheme for ELM (RFWV-ELM) is applied to enhance the classification performance. This ensemble method could solve the instability caused by the randomly assigned weights and biases of original ELM and improve the generalization ability of ELM neural network simultaneously. The experimental results on two benchmark datasets illustrated that the proposed framework combined with CSPM and RFWV-ELM can achieve better classification performance than several existing scene categorization algorithms.

Learning with Information Entropy Method for Transportation Image Retrieval

As a new learning framework, Multi-Instance learning is labeled recently and has successfully found application in vision classification. A novel Multi-instance bag generating method is presented in this paper on basis of Gaussian Mixed Model. The generated GMM model composes not only color but also the locally stable unchangeable components. It is frequently named as MI bag by researchers. Besides, another method called Agglomerative Information Bottleneck clustering is applied to replace the MIL problem with the help of single-instance learning ones. Meanwhile, single-instance classifiers are employed for classification. Finally, ensemble learning is adopted to strengthen classifiers’ generalization ability of RBM (Restricted Boltzmann Machine) as the base classifier. On the basis of large-scale datasets, this method is tested and the corresponding result shows that our method provides high accuracy and good performance for image annotation, feature matching and example-based objectclassification.

Optimal Kullback–Leibler Aggregation via Information Bottleneck

In this paper, we present a method for reducing a regular, discrete-time Markov chain (DTMC) to another DTMC with a given, typically much smaller number of states. The cost of reduction is defined as the Kullback-Leibler divergence rate between a projection of the original process through a partition function and a DTMC on the correspondingly partitioned state space. Finding the reduced model with minimal cost is computationally expensive, as it requires an exhaustive search among all state space partitions, and an exact evaluation of the reduction cost for each candidate partition. Our approach deals with the latter problem by minimizing an upper bound on the reduction cost instead of minimizing the exact cost. The proposed upper bound is easy to compute and it is tight if the original chain is lumpable with respect to the partition. Then, we express the problem in the form of information bottleneck optimization, and propose using the agglomerative information bottleneck algorithm for searching a suboptimal partition greedily, rather than exhaustively. The theory is illustrated with examples and one application scenario in the context of modeling bio-molecular interactions.

Solving Multi-Instance Visual Scene Recognition with Classifier Ensemble Based on Unsupervised Clustering

This paper proposes a new image Multi-Instance (MI) bag generating method, which models an image with a Gaussian Mixed Model (GMM). The generated GMM is treated as an MI bag, of which the color and locally stable invariant components (SIFT) are the instances. Agglomerative Information Bottleneck clustering is employed to transform the MIL problem into single-instance learning problem so that single-instance classifiers can be used for classification. Finally, ensemble learningis involved to further enhance classifiers generalization ability. Experimental results demonstrate that the performance of the proposed framework for image recognition is superior to some common MI algorithms on average in a 5-category scene recognition task Key words:Multi-Instance Learning; Gaussian Mixed Model; AIB Clustering; image modeling; Single-Instance Bag; Ensemble Classifier; Scene Recognition

A Local 3-D Motion Descriptor for Multi-View Human Action Recognition from 4-D Spatio-Temporal Interest Points

In this paper, we address the problem of human action recognition in reconstructed 3-D data acquired by multi-camera systems. We contribute to this field by introducing a novel 3-D action recognition approach based on detection of 4-D (3-D space <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$+$</tex> </formula> time) spatio-temporal interest points (STIPs) and local description of 3-D motion features. STIPs are detected in multi-view images and extended to 4-D using 3-D reconstructions of the actors and pixel-to-vertex correspondences of the multi-camera setup. Local 3-D motion descriptors, histogram of optical 3-D flow (HOF3D), are extracted from estimated 3-D optical flow in the neighborhood of each 4-D STIP and made view-invariant. The local HOF3D descriptors are divided using 3-D spatial pyramids to capture and improve the discrimination between arm- and leg-based actions. Based on these pyramids of HOF3D descriptors we build a bag-of-words (BoW) vocabulary of human actions, which is compressed and classified using agglomerative information bottleneck (AIB) and support vector machines (SVMs), respectively. Experiments on the publicly available i3DPost and IXMAS datasets show promising state-of-the-art results and validate the performance and view-invariance of the approach.

Robust classifying of prokaryotic genomes

In this paper, we propose a method to classify prokaryotic genomes using the agglomerative information bottleneck method for unsupervised clustering. Although the method we present here is closely related to a group of methods based on detecting the presence or absence of genes, our method is different because it uses gene lengths as well. We show that this amended method is reliable. For robustness evaluation, we apply bootstrap and jackknife techniques to input data. As a result, we are able to propose an approach to determine the stability level of a cladogram. We demonstrate that the genome tree produced for a selected small group of genomes looks a lot like a phylogenetic tree of this group.

Compact and adaptive spatial pyramids for scene recognition

Most successful approaches on scene recognition tend to efficiently combine global image features with spatial local appearance and shape cues. On the other hand, less attention has been devoted for studying spatial texture features within scenes. Our method is based on the insight that scenes can be seen as a composition of micro-texture patterns. This paper analyzes the role of texture along with its spatial layout for scene recognition. However, one main drawback of the resulting spatial representation is its huge dimensionality. Hence, we propose a technique that addresses this problem by presenting a compact Spatial Pyramid (SP) representation. The basis of our compact representation, namely, Compact Adaptive Spatial Pyramid (CASP) consists of a two-stages compression strategy. This strategy is based on the Agglomerative Information Bottleneck (AIB) theory for (i) compressing the least informative SP features, and, (ii) automatically learning the most appropriate shape for each category. Our method exceeds the state-of-the-art results on several challenging scene recognition data sets.

Discriminative compact pyramids for object and scene recognition

Spatial pyramids have been successfully applied to incorporating spatial information into bag-of-words based image representation. However, a major drawback is that it leads to high dimensional image representations. In this paper, we present a novel framework for obtaining compact pyramid representation. First, we investigate the usage of the divisive information theoretic feature clustering (DITC) algorithm in creating a compact pyramid representation. In many cases this method allows us to reduce the size of a high dimensional pyramid representation up to an order of magnitude with little or no loss in accuracy. Furthermore, comparison to clustering based on agglomerative information bottleneck (AIB) shows that our method obtains superior results at significantly lower computational costs. Moreover, we investigate the optimal combination of multiple features in the context of our compact pyramid representation. Finally, experiments show that the method can obtain state-of-the-art results on several challenging data sets.