Sequence Of Discrete Symbols Research Articles

Generative Adversarial Networks for Video Summarization Based on Key-frame Selection

Video summarization based on generative adversarial networks (GANs) has been shown to easily produce more realistic results. However, most summary videos are composed of multiple key components. If the selection of some video frames changes during the training process, the information carried by these frames may not be reasonably reflected in the identification results. In this paper, we propose a video summarization method based on selecting keyframes over GANs. The novelty of the proposed method is the discriminator not only identifies the completeness of the video, but also takes into account the value judgment of the candidate keyframes, thus enabling the influence of keyframes on the result value. Given GANs are mainly designed to generate continuous real values, it is generally challenging to generate discrete symbol sequences during the summarization process directly. However, if the generated sample is based on discrete symbols, the slight guidance change of the discrimination network may be meaningless. To better use the advantages of GANs, the study also adopts the video summarization optimization method of GANs under a collaborative reinforcement learning strategy. Experimental results show the proposed method gets a significant summarization effect and character compared with the existing cutting-edge methods.

Detecting renewal states in chains of variable length via intrinsic Bayes factors

Markov chains with variable length are useful parsimonious stochastic models able to generate most stationary sequence of discrete symbols. The idea is to identify the suffixes of the past, called contexts, that are relevant to predict the future symbol. Sometimes a single state is a context, and looking at the past and finding this specific state makes the further past irrelevant. States with such property are called renewal states and they can be used to split the chain into independent and identically distributed blocks. In order to identify renewal states for chains with variable length, we propose the use of Intrinsic Bayes Factor to evaluate the hypothesis that some particular state is a renewal state. In this case, the difficulty lies in integrating the marginal posterior distribution for the random context trees for general prior distribution on the space of context trees, with Dirichlet prior for the transition probabilities, and Monte Carlo methods are applied. To show the strength of our method, we analyzed artificial datasets generated from different models and one example coming from the field of Linguistics.

Music Beat Generation through AI

Music Plays a significant role in our mental health, but composing a song is difficult for composers. While listening to music, Humans can quickly detect and move to beats. Similarly to Language, Music can be represented as a sequence of discrete symbols that form a hierarchical syntax, with notes being roughly like characters and motifs of notes like a word. Several Theoretical and algorithmic approaches have been proposed to account for this ability. Related to, but different from the issue of how we perceive a beat is a question of how we learn to generate and hold a beat. In this paper, we introduce a Music Beat Generator that will interpolate music rhythms and notes that can be used by music artists, Using Artificial intelligence and Machine Learning. In the context of this project, rhythms are one long measure sequence of notes and rest occurring on natural pulse subdivisions of a beat. Music generation using machine learning techniques has been a topic of interest for the past two decades. Music proves to be a different challenge compared to images, among three main dimensions: Firstly, music is temporal, with a hierarchical structure with dependencies across time. Secondly, music consists of interdependent instruments that unfold across time. Thirdly, music is grouped into chords, arpeggios, and melodies – hence each time -step may have multiple outputs. Keywords – Machine Learning; Artificial Intelligence; Music Beats; Algorithm; Chords; Interpolate; Rhythms.

Connecting weighted automata, tensor networks and recurrent neural networks through spectral learning

In this paper, we present connections between three models used in different research fields: weighted finite automata~(WFA) from formal languages and linguistics, recurrent neural networks used in machine learning, and tensor networks which encompasses a set of optimization techniques for high-order tensors used in quantum physics and numerical analysis. We first present an intrinsic relation between WFA and the tensor train decomposition, a particular form of tensor network. This relation allows us to exhibit a novel low rank structure of the Hankel matrix of a function computed by a WFA and to design an efficient spectral learning algorithm leveraging this structure to scale the algorithm up to very large Hankel matrices.We then unravel a fundamental connection between WFA and second-orderrecurrent neural networks~(2-RNN): in the case of sequences of discrete symbols, WFA and 2-RNN with linear activationfunctions are expressively equivalent. Leveraging this equivalence result combined with the classical spectral learning algorithm for weighted automata, we introduce the first provable learning algorithm for linear 2-RNN defined over sequences of continuous input vectors.This algorithm relies on estimating low rank sub-blocks of the Hankel tensor, from which the parameters of a linear 2-RNN can be provably recovered. The performances of the proposed learning algorithm are assessed in a simulation study on both synthetic and real-world data.

Vision-based gesture modeling for signed expressions recognition

A vision-based method for signed expressions recognition consisting of searching for previously prepared word models in the analyzed utterances was developed. The models were created as temporal sequences of grouped descriptors of the place of articulation and the shape of the hand. The descriptors were created using the skeletons determined by the deep neural network and histogram of oriented gradients features. The classification was carried out using the Wagner–Fischer algorithm for determining Levenshtein distance, which was modified to search for fragments in sequences of discrete symbols. Considering the practical application of the developed method, even in the case of partially inaccurate recognition results, it was proposed to visualize the answers in the form of a word cloud. The method was tested for a challenging set of sign language expressions used in the office when applying for an ID card.

Detecting a multigranularity event in an unequal interval time series based on self-adaptive segmenting

Analyzing the temporal behaviors and revealing the hidden rules of objects that produce time series data to detect the events that users are interested in have recently received a large amount of attention. Generally, in various application scenarios and most research works, the equal interval sampling of a time series is a requirement. However, this requirement is difficult to guarantee because of the presence of sampling errors in most situations. In this paper, a multigranularity event detection method for an unequal interval time series, called SSED (self-adaptive segmenting based event detection), is proposed. First, in view of the trend features of a time series, a self-adaptive segmenting algorithm is proposed to divide a time series into unfixed-length segmentations based on the trends. Then, by clustering the segmentations and mapping the clusters to different identical symbols, a symbol sequence is built. Finally, based on unfixed-length segmentations, the multigranularity events in the discrete symbol sequence are detected using a tree structure. The SSED is compared to two previous methods with ten public datasets. In addition, the SSED is applied to the public transport systems in Xiamen, China, using bus-speed time-series data. The experimental results show that the SSED can achieve higher efficiency and accuracy than existing algorithms.

Learning to Communicate and Solve Visual Blocks-World Tasks

We study emergent communication between speaker and listener recurrent neural-network agents that are tasked to cooperatively construct a blocks-world target image sampled from a generative grammar of blocks configurations. The speaker receives the target image and learns to emit a sequence of discrete symbols from a fixed vocabulary. The listener learns to construct a blocks-world image by choosing block placement actions as a function of the speaker’s full utterance and the image of the ongoing construction. Our contributions are (a) the introduction of a task domain for studying emergent communication that is both challenging and affords useful analyses of the emergent protocols; (b) an empirical comparison of the interpolation and extrapolation performance of training via supervised, (contextual) Bandit, and reinforcement learning; and (c) evidence for the emergence of interesting linguistic properties in the RL agent protocol that are distinct from the other two.

Unsupervised Incremental Online Learning and Prediction of Musical Audio Signals

A system is presented that segments, clusters and predicts musical audio in an unsupervised manner, adjusting the number of (timbre) clusters instantaneously to the audio input. A sequence learning algorithm adapts its structure to a dynamically changing clustering tree. The flow of the system is as follows: 1) segmentation by onset detection, 2) timbre representation of each segment by Mel frequency cepstrum coefficients, 3) discretization by incremental clustering, yielding a tree of different sound classes (e.g. instruments) that can grow or shrink on the fly driven by the instantaneous sound events, resulting in a discrete symbol sequence, 4) extraction of statistical regularities of the symbol sequence, using hierarchical N-grams and the newly introduced conceptual Boltzmann machine, and 5) prediction of the next sound event in the sequence. The system's robustness is assessed with respect to complexity and noisiness of the signal. Clustering in isolation yields an adjusted Rand index (ARI) of 82.7% / 85.7% for data sets of singing voice and drums. Onset detection jointly with clustering achieve an ARI of 81.3% / 76.3% and the prediction of the entire system yields an ARI of 27.2% / 39.2%.

Profile HMMs for skeleton-based human action recognition

In this paper, a novel approach based Hidden Markov Models (HMMs) approach is proposed for human action recognition using 3D positions of body joints. Unlike existing works, this paper addresses the challenging problem of spatio-temporal alignment of human actions which come from intra-class variability and inter-class similarity of actions. The first and foremost actions are segmented into meaningful action-units called dynamic instants and intervals by using motion velocities, the direction of motion, and the curvatures of 3D trajectories. Then action-units with its spatio-temporal feature sets are clustered using unsupervised learning, like Self-Organizing Mapping (SOM), to generate a sequence of discrete symbols. To overcome an abrupt change or an abnormal in its gesticulation between different appearances of the same kind of action, profile HMMs are applied with these symbol sequences using Viterbi and Baum–Welch algorithms for human activity recognition. The effectiveness of the proposed method is evaluated on three challenging 3D action datasets captured by commodity depth cameras. The experimental evaluations show that the proposed approach achieves promising results compared to other state-of-the-art algorithms.

Human-computer music performance: A brief history and future prospects

Computer accompaniment began in the eighties as a technology to synchronize computers to live musicians by sensing, following, and adapting to expressive musical performances. The technology has progressed from systems where performances were modeled as sequences of discrete symbols, i.e., pitches, to modern systems that use continuous probabilistic models. Although score following techniques have been a common focus, computer accompaniment research has addressed many other interesting topics, including the musical adjustment of tempo, the problem of following an ensemble of musicians, and making systems more robust to unexpected mistakes by performers. Looking toward the future, we find that score following is only one of many ways musicians use to synchronize. Score following is appropriate when scores exist and describe the performance accurately, and where timing deviations are to be followed rather than ignored. In many cases, however, especially in popular music forms, tempo is rather steady, and performers improvise many of their parts. Traditional computer accompaniment techniques do not solve these important music performance scenarios. The term Human-Computer Music Performance (HCMP) has been introduced to cover a broader spectrum of problems and technologies where humans and computers perform music together, adding interesting new problems and directions for future research.

Exactly solvable chaos in an electromechanical oscillator

A novel electromechanical chaotic oscillator is described that admits an exact analytic solution. The oscillator is a hybrid dynamical system with governing equations that include a linear second order ordinary differential equation with negative damping and a discrete switching condition that controls the oscillatory fixed point. The system produces provably chaotic oscillations with a topological structure similar to either the Lorenz butterfly or Rössler's folded-band oscillator depending on the configuration. Exact solutions are written as a linear convolution of a fixed basis pulse and a sequence of discrete symbols. We find close agreement between the exact analytical solutions and the physical oscillations. Waveform return maps for both configurations show equivalence to either a shift map or tent map, proving the chaotic nature of the oscillations.

On the Structure of Real-Time Encoding and Decoding Functions in a Multiterminal Communication System

A real-time communication system with two encoders communicating with a single receiver over separate noisy channels is considered. The two encoders make distinct partial observations of a Markov source. Each encoder must encode its observations into a sequence of discrete symbols. The symbols are transmitted over noisy channels to a finite memory receiver that attempts to reconstruct some function of the state of the Markov source. Encoding and decoding must be done in real-time, that is, the distortion measure does not tolerate delays. Under the assumption that the encoders' observations are conditionally independent Markov chains given an unobserved time-invariant random variable, results on the structure of optimal real-time encoding and decoding functions are obtained. It is shown that there exist finite-dimensional sufficient statistics for the encoders. The problem with noiseless channels and perfect memory at the receiver is then considered. A new methodology to find the structure of optimal real-time encoders is employed. A sufficient statistic with a time-invariant domain is found for this problem. This methodology exploits the presence of common information between the encoders and the receiver when communication is over noiseless channels.

Entropy estimation using pattern matching in bioacoustic signals.

Many bioacoustic signals consist of a sequence of discrete stereotyped sounds occurring in repeated patterns. A natural question to ask is how best to characterize the underlying structure of the source producing the sequence of sounds. The structure of the source manifests itself as constraints on the patterns observed in the sequence of sounds. These constraints determine how predictable the order of the sounds is. The information entropy of a discrete symbol sequence is a quantitative measure of how unpredictable the sequence is. A straightforward but biased technique for estimating the entropy of an unknown source is to substitute observed symbol frequencies into parametric models such as Markov models. More general nonparametric entropy estimators exploit the relationship between the entropy and the average length of matching patterns within the sequence. This nonparametric entropy estimate forms an upper bound on the amount of information conveyed by the sequence of sounds. Additionally, comparing entropy estimates from the parametric and nonparametric models provides a hypothesis test determining whether the parametric model sufficiently captures the constraints of the source. These techniques are illustrated in analyses of humpback whale songs and leopard seal calling bouts.

Anomaly Detection and Diagnosis Algorithms for Discrete Symbol Sequences with Applications to Airline Safety

We present a set of novel algorithms which we call sequenceMiner that detect and characterize anomalies in large sets of high-dimensional symbol sequences that arise from recordings of switch sensors in the cockpits of commercial airliners. While the algorithms that we present are general and domain-independent, we focus on a specific problem that is critical to determining the system-wide health of a fleet of aircraft. The approach taken uses unsupervised clustering of sequences using the normalized length of the longest common subsequence as a similarity measure, followed by detailed outlier analysis to detect anomalies. In this method, an outlier sequence is defined as a sequence that is far away from the cluster center. We present new algorithms for outlier analysis that provide comprehensible indicators as to why a particular sequence is deemed to be an outlier. The algorithms provide a coherent description to an analyst of the anomalies in the sequence when compared to more normal sequences. In the final section of the paper, we demonstrate the effectiveness of sequenceMiner for anomaly detection on a real set of discrete-sequence data from a fleet of commercial airliners. We show that sequenceMiner discovers actionable and operationally significant safety events. We also compare our innovations with standard hidden Markov models, and show that our methods are superior.

Learning Atomic Human Actions Using Variable-Length Markov Models

Visual analysis of human behavior has generated considerable interest in the field of computer vision because of its wide spectrum of potential applications. Human behavior can be segmented into atomic actions, each of which indicates a basic and complete movement. Learning and recognizing atomic human actions are essential to human behavior analysis. In this paper, we propose a framework for handling this task using variable-length Markov models (VLMMs). The framework is comprised of the following two modules: a posture labeling module and a VLMM atomic action learning and recognition module. First, a posture template selection algorithm, based on a modified shape context matching technique, is developed. The selected posture templates form a codebook that is used to convert input posture sequences into discrete symbol sequences for subsequent processing. Then, the VLMM technique is applied to learn the training symbol sequences of atomic actions. Finally, the constructed VLMMs are transformed into hidden Markov models (HMMs) for recognizing input atomic actions. This approach combines the advantages of the excellent learning function of a VLMM and the fault-tolerant recognition ability of an HMM. Experiments on realistic data demonstrate the efficacy of the proposed system.

Information entropy of humpback whale songs

The structure of humpback whale (Megaptera novaeangliae) songs was examined using information theory techniques. The song is an ordered sequence of individual sound elements separated by gaps of silence. Song samples were converted into sequences of discrete symbols by both human and automated classifiers. This paper analyzes the song structure in these symbol sequences using information entropy estimators and autocorrelation estimators. Both parametric and nonparametric entropy estimators are applied to the symbol sequences representing the songs. The results provide quantitative evidence consistent with the hierarchical structure proposed for these songs by Payne and McVay [Science 173, 587-597 (1971)]. Specifically, this analysis demonstrates that: (1) There is a strong structural constraint, or syntax, in the generation of the songs, and (2) the structural constraints exhibit periodicities with periods of 6-8 and 180-400 units. This implies that no empirical Markov model is capable of representing the songs' structure. The results are robust to the choice of either human or automated song-to-symbol classifiers. In addition, the entropy estimates indicate that the maximum amount of information that could be communicated by the sequence of sounds made is less than 1 bit per second.

Bayesian blind and semi-blind equalisation of channels with Markov inputs

An original full Bayesian approach is developed for blind and semi-blind equalisation of fading channels with Markov inputs. The sequence of discrete symbols is estimated according to a marginal maximum a posteriori criterion; the other unknown parameters are regarded as random nuisance parameters and are integrated out analytically. A batch algorithm is proposed to maximise the marginal posterior distribution. Simulation results are presented to demonstrate the effectiveness of the method.

Categorical perception and the convergence of social learning

About 50 years ago, Alvin Liberman and his colleagues at Haskins Laboratories discovered ‘‘categorical perception’’ of certain phonetic dimensions, where discrimination is hardly better than identification. They proposed this as one piece of converging evidence for a special perceptual mode for speech, part of a species-specific evolutionary adaptation for articulate language. This special sensory-motor module, they argued, makes human speech so uniquely efficient at transmitting discrete symbol sequences in sound. Lively arguments about all aspects of this reasoning continue to this day. This paper broadens the discussion beyond perception in individuals, by suggesting a role for categorical perception in enabling communities of speakers to form and maintain consensus about the pronunciation of tens of thousands of morphemes. Through computer simulations, simple and plausible assumptions will be shown under which the ‘‘pronouncing dictionaries’’ of the members of a speech community will converge rapidly from random starting points. Among these assumptions, a form of categorical perception plays a key role. With it, simulations converge to a consensus from which deviations occur rarely and hardly ever spread. Without it, individual pronunciation beliefs wander chaotically across time in the phonetic space, and at a given time, differences in belief increase rapidly with social distance. The role of partly categorical perception, as in so-called ‘‘magnet effects,’’ will also be addressed.

Avoiding ill-convergence of finite dimensional blind adaptation schemes excited by discrete symbol sequences

A general framework for convergence analysis of finite-dimensional blind adaptation algorithms of Bussgang type is presented. The approach allows discrete symbol sets and can be used for analysis of systems with both poles and zeros. The main tool of analysis is an associated differential equation whose stability properties are proved to be tied to the convergence properties of a general blind stochastic approximation algorithm. The recently highlighted ill-convergence problem of for example the constant modulus algorithm (CMA) is then addressed. The problem is partially solved using new blind adaptation algorithms which are not derived by criterion minimization. Instead, the averaged updating directions of the suggested stochastic gradient schemes are designed to guarantee global stability of the associated differential equations.

Identifying Hierarchical Structure in Sequences: A linear-time algorithm

SEQUITUR is an algorithm that infers a hierarchical structure from a sequence of discrete symbols by replacing repeated phrases with a grammatical rule that generates the phrase, and continuing this process recursively. The result is a hierarchical representation of the original sequence, which offers insights into its lexical structure. The algorithm is driven by two constraints that reduce the size of the grammar, and produce structure as a by-product. SEQUITUR breaks new ground by operating incrementally. Moreover, the method's simple structure permits a proof that it operates in space and time that is linear in the size of the input. Our implementation can process 50,000 symbols per second and has been applied to an extensive range of real world sequences.