Efficient discovery of co-movement patterns from video data

In image and video data, visual pattern refers to re‐occurring composition of visual primitives. Such visual patterns extract the essence of the image and video data that convey rich information. However, unlike frequent patterns in transaction data, there are considerable visual content variations and complex spatial structures among visual primitives, which make effective exploration of visual patterns a challenging task. Many methods have been proposed to address the problem of visual pattern discovery during the past decade. In this article, we provide a review of the major progress in visual pattern discovery. We categorize the existing methods into two groups: bottom‐up pattern discovery and top‐down pattern modeling. The bottom‐up pattern discovery method starts with unordered visual primitives followed by merging the primitives until larger visual patterns are found. In contrast, the top‐down method starts with the modeling of visual primitive compositions and then infers the pattern discovery result. A summary of related applications is also presented. At the end we identify the open issues for future research.WIREs Data Mining Knowl Discov2014, 4:24–37. doi: 10.1002/widm.1110This article is categorized under:Algorithmic Development > MultimediaAlgorithmic Development > Structure Discovery

A major source of revenue shrink in retail stores is the intentional or unintentional failure of proper checking out of items by the cashier. More recently, a few automated surveillance systems have been developed to monitor cashier lanes and detect non-compliant activities such as fake item checkouts or scans done with the intention of deriving monetary benefit. These systems use data from surveillance video cameras and transaction logs (TLog) recorded at the Point-of-Sale (POS). In this paper, we present a pattern discovery based approach to detect fraudulent events at the POS. Our approach is based on mining time-ordered text streams, representing retail transactions, formed from a combination of visually detected checkout related activities called primitives and barcodes from TLog data. Patterns representing single item checkouts, i.e. anchored around a single barcode, are discovered from these text streams using an efficient pattern discovery technique called Teiresias. The discovered patterns are used to build models for true and fake item scans by retaining or discarding the anchoring barcodes in those patterns respectively. A pattern matching and classification scheme is designed to robustly detect non-compliant cashier activities in the presence of noise in either the TLog or the video data. Different weighting schemes for quantifying the relative importance of the discovered patterns are explored: Frequency, Support Vector Machine (SVM) and Frequency+SVM. Using a large scale dataset recorded from retail stores, our approach discovers semantically meaningful cashier scan patterns. Our experiments also suggest that different weighting schemes result in varied false and true positive performances on the task of fake scan detection.

This minitrack encompasses papers of a quantitative, theoretical or applied nature that focus on: Content Mining of Social Media -- discovery of patterns from the text, images, audio, video and other data generated by Social Media sites Structure Mining of Social Media -- social network analysis of the node and connection (graph) structures underlying Social Media sites

BackgroundPolypeptides are composed of amino acids covalently bonded via a peptide bond. The majority of peptide bonds in proteins is found to occur in the trans conformation. In spite of their infrequent occurrence, cis peptide bonds play a key role in the protein structure and function, as well as in many significant biological processes.ResultsWe perform a systematic analysis of regions in protein sequences that contain a proline cis peptide bond in order to discover non-random associations between the primary sequence and the nature of proline cis/trans isomerization. For this purpose an efficient pattern discovery algorithm is employed which discovers regular expression-type patterns that are overrepresented (i.e. appear frequently repeated) in a set of sequences. Four types of pattern discovery are performed: i) exact pattern discovery, ii) pattern discovery using a chemical equivalency set, iii) pattern discovery using a structural equivalency set and iv) pattern discovery using certain amino acids' physicochemical properties. The extracted patterns are carefully validated using a specially implemented scoring function and a significance measure (i.e. log-probability estimate) indicative of their specificity. The score threshold for the first three types of pattern discovery is 0.90 while for the last type of pattern discovery 0.80. Regarding the significance measure, all patterns yielded values in the range [-9, -31] which ensure that the derived patterns are highly unlikely to have emerged by chance. Among the highest scoring patterns, most of them are consistent with previous investigations concerning the neighborhood of cis proline peptide bonds, and many new ones are identified. Finally, the extracted patterns are systematically compared against the PROSITE database, in order to gain insight into the functional implications of cis prolyl bonds.ConclusionCis patterns with matches in the PROSITE database fell mostly into two main functional clusters: family signatures and protein signatures. However considerable propensity was also observed for targeting signals, active and phosphorylation sites as well as domain signatures.

Videos are ubiquitous and have significantly impacted our communication and information consumption. The video, as data, has helped researchers understand how human interactions and relationships develop and change, and how patterns emerge in various circumstances and interpretations. Given the expanding relevance of video data in social science and medical research and the constant introduction of new formats and sources, it is critical to be able to conduct a thorough analysis of this multimodal data. However, the few methodologies (e.g., Actor Network Theory, Picture Theory) appropriate to video data analysis lack detailed guidelines on how to select, organize, and examine the multimodality of video data. This article aims to overcome this practice or methodological gap by proposing and demonstrating the Visual-Verbal Video Analysis (VVVA) method, a six-step framework adapted from Multimodal Theory and Visual Grounded Theory for organizing and evaluating video material according to the following dimensions: general characteristics of the video; multimodal characteristics; visual characteristics; characteristics of primary and secondary characters; and content and compositional characteristics including the transmission of messages, emotions, and discourses. This article also looks at the theories underlying video data analysis, focusing on Grounded Theory and Multimodality Theory, and provides multiple examples of coding and interpretive processes to deepen understanding and comprehension. The VVVA data extraction matrices provide a systematic coding approach for verbal, visual, and textual content, allowing for structured, coherent extraction that supports the discovery of patterns and links among disparate types of information. The VVVA method may be applied to a wide range of video data in social and medical sciences that vary in length and originate from different sources (e.g., open access web sources, pre-recorded organizational videos and recordings created for research purposes). The VVVA method effectively tracks the ongoing research process, and can manage data sets of various sizes.

Pattern discovery is a core data mining activity. Initial approaches were dominated by the frequent pattern discovery paradigm -- only patterns that occur frequently in the data were explored. Having been thoroughly researched and its limitations now well understood, this paradigm is giving way to a new one, which can be called statistically sound pattern discovery. In this paradigm, the main impetus is to discover statistically significant patterns, which are unlikely to have occurred by chance and are likely to hold in future data. Thus, the new paradigm provides a strict control over false discoveries and overfitting.This tutorial covers both classic and cutting-edge research topics on pattern discovery combined to statistical significance testing. We start with an advanced introduction to the relevant forms of statistical significance testing, including different schools and alternative models, their underlying assumptions, practical issues, and limitations. We then discuss their application to data mining specific problems, including evaluation of nested patterns, the multiple testing problem, algorithmic strategies and real-world considerations. We present the current state-of-the art solutions and explore in detail how this approach to pattern discovery can deliver efficient and effective discovery of small sets of interesting patterns.

TV ratings are a widely used indicator in the TV broadcasting field. While TV ratings are mainly used in advertising, they can also be used as a social sensor that reflects the interests of people. This paper presents a framework for discovering audience behavior through the mining of TV ratings. We have established a framework that enables discovery of numerous patterns of audience behavior from TV ratings. Used along with other multimedia contents such as video and text, it enables various types of knowledge to be semi-automatically found, such as what types of news programs are of most interest and what are the key visual features for acquiring high TV ratings. The discovery of audience behavior is achieved by focusing on the change points in the rating data, i.e., the points in time where many people switch the channel or turn the television on or off. Rich descriptions that characterize these points are extracted from multimedia contents, and then various filtering techniques are used to extract specific patterns of interest. Several applications of this framework for discovering knowledge demonstrated that it can effectively extract various types of audience behavior. To the best of our knowledge, this work is the first work to analyze the use of ratings data in combination with video and other multimedia data.

Pattern discovery is an essential computational music analysis method for revealing intra-opus repetition and inter-opus recurrence. This chapter applies pattern discovery to a corpus of songs for the bagana, a large lyre played in Ethiopia. An important and unique aspect of this repertoire is that frequent and rare motifs have been explicitly identified and used by a master bagana teacher in Ethiopia. A new theorem for pruning of statistically under-represented patterns from the search space is used within an efficient pattern discovery algorithm. The results of the chapter show that over- and under-represented patterns can be discovered in a corpus of bagana songs, and that the method can reveal with high significance the known bagana motifs of interest.

Mining multi-cellular spatial patterns associated with biological events from high-resolution spatial omics data remains a fundamental challenge. While current computational methods have advanced from pairwise associations to identifying higher-order spatial domains, they often lack the granularity to resolve subtle local architectural shifts or the statistical framework to quantify condition-specificity. Here, we present FDPMining (Frequent and Distinctive spatial Patterns Mining), a computational framework that reformulates the biological problem of pattern discovery into a scalable data mining task through a Neighborhood-to-Sequence (N2S) encoding strategy. This transformation uniquely converts spatial grid neighborhoods for each cell into lossless and reversible numerical sequences, enabling efficient and scalable discovery of FDPs (Frequent and Distinctive spatial Patterns) via data mining algorithms. Our approach systematically explores the vast combinatorial space of cellular arrangements to identify FDPs associated with specific biological conditions. To enable spatial traceability, we further develop FDPs-Mapping, a spatial reconstruction component that maps identified patterns back to their original tissue context. This advancement allows researchers to examine and interpret patterns directly in situ. In extensive benchmarking, FDPMining demonstrates superior sensitivity in capturing subtle and condition-specific differences, outperforming state-of-the-art pairwise and higher-order pattern discovery methods. We applied our framework across diverse biological systems and spatial omics technologies, successfully identifying biologically meaningful spatial multicellular patterns in axolotl brain regeneration, brain aging, liver zonation, Alzheimer’s disease, and colorectal cancer. Notably, FDPMining enables landmark-anchored pattern discovery around specific anatomical or pathological features such as blood vessels or amyloid plaques, among which applications to Alzheimer’s disease revealed previously inaccessible insights into the multicellular organization of these microenvironments. FDPMining offers a paradigm for quantitatively dissecting spatial heterogeneity in complex tissues, enabling more systematic mining, visualization, and interpretation of cellular organization across diverse biological conditions.

Thanks to recent prevalence of location sensors, collecting massive spatiotemporal datasets containing moving object trajectories has become possible, providing an exceptional opportunity to derive interesting insights about behavior of the moving objects such as people, animals, and vehicles. In particular, mining patterns from co-movements of objects (such as players of a sports team, joints of a person while walking, and cars in a transportation network) can lead to the discovery of interesting patterns (e.g., offense tactics of the sports team, gait signature of the person, and driving behaviors causing heavy traffic). With our prior work, we proposed efficient algorithms to mine frequent co-movement patterns from trajectory datasets. In this paper, we focus on the problem of efficient query processing on massive co-movement pattern datasets generated by such pattern mining algorithms. Given a dataset of frequent co-movement patterns, various spatiotemporal queries can be posed to retrieve relevant patterns among all generated patterns from the pattern dataset. We term such queries “pattern queries”. Co-movement patterns are often numerous due to combinatorial complexity of such patterns, and therefore, co-movement pattern datasets grow very large, rendering naive execution of the pattern queries ineffective. In this paper, we propose novel index structures and query processing algorithms for efficient answering of two families of range pattern queries on massive co-movement pattern datasets, namely, spatial range pattern queries and temporal range pattern queries. Our extensive empirical studies with three real datasets have demonstrated the efficiency of the proposed methods.

Changes in currency exchange rates can bring significant impact to international macroeconomics and vice versa. Discovering co-movement patterns of currencies in a specific period can help explain economic and financial relationships between the countries under the flexible exchange rate system. In order to discover various co-movement patterns of foreign exchange rates, we propose an evolutionary biclustering algorithm to search for the biclusters each of which has a subset of rows (time points) and a subset of columns (currencies), indicating a group of currencies co-moved within a specific period. We test this algorithm on the historical monthly data of 15 currencies, and successfully detect a number of co-movement patterns. We also find that the discovered patterns will be useful as a guide for foreign currency investment.

The advent of GPS technologies generates location data-streams and accentuates the importance of developing practical tools that can process and analyze the vast amounts of location data at a given moment in a meaningful way. Profiling the trajectory of a moving object with respect to the trajectories of its surrounding objects, for example, can elicit its mobility behaviour and analyze it in order to inform domain experts with critical knowledge in real time. For instance, clustering multiple moving objects with respect to their spatial and temporal dimension to identify co-movement patterns. In this paper, we propose a novel graph-based online co-movement pattern mining algorithm, called EvolvingClusters, which can be used to discover different collective movement behaviours (like the well-known flocks and convoys) in a unified way based on the activity of multiple concurrent objects through time and space. We evaluate EvolvingClusters using real-world and synthetic datasets from multiple mobility domains. Our study demonstrates the effectiveness of the proposed algorithm as well as its value towards a tool to profile semantically rich behaviour and with capabilities to observe and categorize multiple moving objects in real-time.

Thanks to recent prevalence of location sensors, collecting massive spatiotemporal datasets containing moving object trajectories has become possible, providing an exceptional opportunity to derive interesting insights about behavior of the moving objects such as people, animals, and vehicles. In particular, mining patterns from co-movements of objects (such as movements by players of a sports team, joints of a person while walking, and cars in a transportation network) can lead to the discovery of interesting patterns (e.g., offense tactics of a sports team, gait signature of a person, and driving behaviors causing heavy traffic). Given a dataset of frequent co-movement patterns, various spatial and spatiotemporal queries can be posed to retrieve relevant patterns among all generated patterns from the pattern dataset. We term such queries, pattern queries. Co-movement patterns are often numerous due to combinatorial complexity of such patterns, and therefore, co-movement pattern datasets often grow very large in size, rendering naive execution of the pattern queries ineffective. In this paper, we propose the FCPIR framework, which offers a variety of index structures for efficient answering of various range pattern queries on massive co-movement pattern datasets, namely, spatial range pattern queries, temporal range (time-slice) pattern queries, and spatiotemporal range pattern queries.

Sequential pattern mining is a challenging issue because of the high complexity of temporal pattern discovering from numerous sequences. Current mining approaches either require frequent database scanning or the generation of several intermediate databases. As databases may fit into the ever-increasing main memory, efficient memory-based discovery of sequential patterns is becoming possible. In this paper, we propose a memory indexing approach for fast sequential pattern mining, named MEMISP. During the whole process, MEMISP scans the sequence database only once to read data sequences into memory. The find-then-index technique is recursively used to find the items that constitute a frequent sequence and constructs a compact index set which indicates the set of data sequences for further exploration. As a result of effective index advancing, fewer and shorter data sequences need to be processed in MEMISP as the discovered patterns get longer. Moreover, we can estimate the maximum size of the total memory required, which is independent of the minimum support threshold, in MEMISP. Experimental results indicate that MEMISP outperforms both GSP and PrefixSpan (general version) without the need for either candidate generation or database projection. When the database is too large to fit into memory in a batch, we partition the database, mine patterns in each partition, and validate the true patterns in the second pass of database scanning. Experiments performed on extra-large databases demonstrate the good performance and scalability of MEMISP, even with very low minimum support. Therefore, MEMISP can efficiently mine sequence databases of any size, for any minimum support values.

Mining sequential patterns is an important issue for the complexity of temporal pattern discovering from sequences. Current mining approaches either require many times of database scanning or generate several intermediate databases. As databases may fit into the ever-increasing main memory, efficient memory-based discovery of sequential patterns will become possible. In this paper, we propose a memory indexing approach for fast sequential pattern mining, named MEMISP. During the whole process, MEMISP scans the sequence database only once for reading data sequences into memory. The find-then- index technique recursively finds the items which constitute a frequent sequence and constructs a compact index set which indicates the set of data sequences for further exploration. Through effective index advancing, fewer and shorter data sequences need to be processed in MEMISP as the discovered patterns getting longer. Moreover, the maximum size of total memory required, which is independent of minimum support threshold in MEMISP, can be estimated. The experiments indicates that MEMISP outperforms both GSP and PrefixSpan algorithms. MEMISP also has good linear scalability even with very low minimum support. When the database is too large to fit in memory in a batch, we partition the database, mine patterns in each partition, and validate the true patterns in the second pass of database scanning. Therefore, MEMISP may efficiently mine databases of any size, for any minimum support values.