Temporal Stream Research Articles

Anton Arensky — educator and master of musical form

The article researches the interaction between polyphonic and homophonic and harmonic principles of form in the genre of lyrical miniature. Anton Arensky in his “Manual to Studying Forms of Instrumental and Vocal Music” (the first in Russia, 1893–894), sticks to the common opposition of counterpoint and homophonic building of form. However, in his artistic practice he follows the harmonic interpretation of imitational constructions. Canon as one of symbols of polyphonic reasoning is subject to patterns of homophonic song forms in the age of romanticism. Accepting the alien symmetrical time arrangement (with its dynamic effect of extrapolation) and the directing action of tonal functionality, it becomes not the basic but the additional factor of musical building. The vertical correlation of parts (its essence is in transmission of the same material from one voice into another), which is typical of imitational structure, ceases to be principal for form. Functionally differential, horizontal, juxtaposition when initial exposition is followed by the thematic and harmonic development with subsequent final comes to the fore. Temporal independence of parts, characteristic of a genuinely linear canon, almost vanishes in an integrated temporal stream which is set by the rhythm of harmonic changes. The homophonic (“horizontal”) form becomes leading, while the polyphonic (“vertical”) form moves back to the level of texture. There is a possibility of a different combination of polyphonic and homophonic regularities in composition: adding to the given«cantus» (in the form of basso ostinato in the work by Arensky) not a contrapuntal part, or parts, but the whole homophonic form with its principle of organization; and the resulting emergence of the counterpoint of forms (according to Y. Kholopov), the origin of which has its roots in the Baroque music.

Towards efficient and objective work sampling: Recognizing workers' activities in site surveillance videos with two-stream convolutional networks

Abstract Capturing the working states of workers on foot allows managers to precisely quantify and benchmark labor productivity, which in turn enables them to evaluate productivity losses and identify causes. Work sampling is a widely used method for this task, while suffers from low efficiency as only one worker is selected for each observation. Attentional selection asymmetry can also bias its uniform object selection assumption. Existing vision-based methods are primarily oriented towards recognizing single, separated activities involving few workers or equipment. In this paper, we introduce an activity recognition method, which receives surveillance videos as input and produces diverse and continuous activity labels of individual workers in the field of view. Convolutional networks are used to recognize activities, which are encoded in spatial and temporal streams. A new fusion strategy is developed to combine the recognition results of the two streams. The experimental results show that our activity recognition method has achieved an average accuracy of 80.5%, which is comparable with the state-of-the-art of activity recognition in the computer vision community, given the severe camera motion and low resolution of site surveillance videos and the marginal inter-class difference and significant intra-class variation of workers' activities. We also demonstrate that our method can underpin the implementation of efficient and objective work sampling. The training and test datasets of the study are publicly available.

Natural echolocation sequences evoke echo-delay selectivity in the auditory midbrain of the FM bat, Eptesicus fuscus.

Echolocating bats must process temporal streams of sonar sounds to represent objects along the range axis. Neuronal echo-delay tuning, the putative mechanism of sonar ranging, has been characterized in the inferior colliculus (IC) of the mustached bat, an insectivorous species that produces echolocation calls consisting of constant frequency and frequency modulated (FM) components, but not in species that use FM signals alone. This raises questions about the mechanisms that give rise to echo-delay tuning in insectivorous bats that use different signal designs. To investigate whether stimulus context may account for species differences in echo-delay selectivity, we characterized single-unit responses in the IC of awake passively listening FM bats, Eptesicus fuscus, to broadcasts of natural sonar call-echo sequences, which contained dynamic changes in signal duration, interval, spectrotemporal structure, and echo-delay. In E. fuscus, neural selectivity to call-echo delay emerges in a population of IC neurons when stimulated with call-echo pairs presented at intervals mimicking those in a natural sonar sequence. To determine whether echo-delay selectivity also depends on the spectrotemporal features of individual sounds within natural sonar sequences, we studied responses to computer-generated echolocation signals that controlled for call interval, duration, bandwidth, sweep rate, and echo-delay. A subpopulation of IC neurons responded selectively to the combination of the spectrotemporal structure of natural call-echo pairs and their temporal patterning within a dynamic sonar sequence. These new findings suggest that the FM bat's fine control over biosonar signal parameters may modulate IC neuronal selectivity to the dimension of echo-delay. NEW & NOTEWORTHY Echolocating bats perform precise auditory temporal computations to estimate their distance to objects. Here, we report that response selectivity of neurons in the inferior colliculus of a frequency modulated bat to call-echo delay, or target range tuning, depends on the temporal patterning and spectrotemporal features of sound elements in a natural echolocation sequence. We suggest that echo responses to objects at different distances are gated by the bat's active control over the spectrotemporal patterning of its sonar emissions.

Global Temporal Representation Based CNNs for Infrared Action Recognition

Infrared human action recognition has many advantages, i.e., it is insensitive to illumination change, appearance variability, and shadows. Existing methods for infrared action recognition are either based on spatial or local temporal information, however, the global temporal information, which can better describe the movements of body parts across the whole video, is not considered. In this letter, we propose a novel global temporal representation named optical-flow stacked difference image (OFSDI) and extract robust and discriminative feature from the infrared action data by considering the local, global, and spatial temporal information together. Due to the small size of the infrared action dataset, we first apply convolutional neural networks on local, spatial, and global temporal stream respectively to obtain efficient convolutional feature maps from the raw data rather than train a classifier directly. Then these convolutional feature maps are aggregated into effective descriptors named three-stream trajectory-pooled deep-convolutional descriptors by trajectory-constrained pooling. Furthermore, we improve the robustness of these features by using the locality-constrained linear coding (LLC) method. With these features, a linear support vector machine (SVM) is adopted to classify the action data in our scheme. We conduct the experiments on infrared action recognition datasets InfAR and NTU RGB+D. The experimental results show that the proposed approach outperforms the representative state-of-the-art handcrafted features and deep learning features based methods for the infrared action recognition.

‘It's not cricket’: Financial time and postcolonial temporalities in Joseph O'Neill's ‘Netherland’

AbstractBased on a reading of Joseph O'Neill's 2008 novel Netherland, this article discusses the relationship between cricket and finance capitalism from the perspective of time and temporality. Despite its function as a global commodity, cricket inserts a flow of postcolonial time into the temporal streams of transnational market culture, neoliberalism, and the increasing financialization of the world. Set in the aftermath of 9/11 and before the financial crisis of 2008, Netherland juxtaposes the deviant temporal power of cricket with the time structures of finance capitalism to illustrate how the conduct of Wall Street before the crisis can be understood as a colonial appropriation. In O'Neill's novel, this conflict is embodied in the precarious friendship of a cosmopolitan Dutch financial analyst and a Trinidadian version of Jay Gatsby.

The consistency of the subjective concept of randomness.

A pervasive bias in the subjective concept of randomness is that people often expect random sequences to exhibit more alternations than produced by genuine random processes. What is less known is the stability of this bias. Here, we examine two important aspects of the over-alternation bias: first, whether this bias is present in stimuli that vary across feature dimensions, sensory modalities, presentation modes and probing methods, and, second, how consistent the bias is across these stimulus variations. In Experiment 1, participants adjusted sequences until they looked maximally random. The sequences were presented as temporal streams of colors, shapes, auditory tones or tiled as spatial matrices. In Experiment 2, participants produced random matrices by adjusting the color of each cell. We replicated the findings using a within-subjects design in Experiment 3. We found that participants judged and produced over-alternating stimuli as the most random. Importantly, this bias was consistent across presentation modes (temporal vs spatial), feature dimensions (color vs shape), sensory modalities (visual vs auditory), speed (fast vs slow), stimulus size (small vs large matrices) and probing methods (adjusting the generating process vs individual bits). Overall, the results suggest that the subjective concept of randomness is highly stable across stimulus variations.

The plurality of temporal reckoning among the Maya

This paper presents an overview of time reckoning in several domains of Maya language and culture, as observed in the Sierra region of Yucatan in the last decades of the twentieth century. It demonstrates that multiple systems of temporal reckoning and orientation co-operate in the traditional domains of daily practice, and it attempts to formulate principles by which different systems are combined in actional frameworks It is argued that most cyclicity in contemporary Yucatec Maya is derivative of natural or social processes and that in rituals it results from the translation of spatial arrays into temporal sequences. The paper tracks the varieties of cyclic and spiral time measurement in several domains, where different units and levels of temporal measurement and different rhythms are observed: 1) the field of co-presence in which utterances are performed, including deictic time; 2) the diurnal cycle and its actional correlates; 3) the agricultural cycles, the forest life cycle and the labor they imply; 4) domestic space whose time involves in addition the age and generational relations among co-residents; 5) ritual time in Yucatec shamanic practice, which is given special attention. Ritual practices display the most dramatic cyclicity, compounded by sedimentation, deictic time, historical time and cosmological space. Their chronometric dimensions are so elaborate that rituals can be considered a time machine. Viewed from practice, there is no single modality of Maya time, but a diachronic synchronization of multiple temporal streams which produces time as the variable product of tzol reproduction and meyah ‘work.’

Semantic Scene Mapping with Spatio-temporal Deep Neural Network for Robotic Applications

Semantic scene mapping is a challenge and significant task for robotic application, such as autonomous navigation and robot-environment interaction. In this paper, we propose a semantic pixel-wise mapping system for potential robotic applications. The system includes a novel spatio-temporal deep neural network for semantic segmentation and a Simultaneous Localisation and Mapping (SLAM) algorithm for 3D point cloud map. Their combination yields a 3D semantic pixel-wise map. The proposed network consists of Convolutional Neural Networks (CNNs) with two streams: spatial stream with images as the input and temporal stream with image differences as the input. Due to the use of both spatial and temporal information, it is called spatio-temporal deep neural network, which shows a better performance in both accuracy and robustness in semantic segmentation. Further, only keyframes are selected for semantic segmentation in order to reduce the computational burden for video streams and improve the real-time performance. Based on the result of semantic segmentation, a 3D semantic map is built up by using the 3D point cloud map from a SLAM algorithm. The proposed spatio-temporal neural network is evaluated on both Cityscapes benchmark (a public dataset) and Essex Indoor benchmark (a dataset we labelled ourselves manually). Compared with the state-of-the-art spatial only neural networks, the proposed network achieves better performances in both pixel-wise accuracy and Intersection over Union (IoU) for scene segmentation. The constructed 3D semantic map with our methods is accurate and meaningful for robotic applications.

More efficient and effective tricks for deep action recognition

Deep convolutional network has achieved great success in visual recognition of static images, while it is not so advantageous as traditional methods in action recognition in videos. As two-stream-style convolutional network gaining best performance in human action recognition, there exist obstacles such as selecting different pre-train models and hyper-parameters, and high computation consumption. In this paper, we propose two efficient and effective methods for action recognition, based on two-stream convolutional network. (1) Reducing computational cost of temporal stream while achieving the same accuracy, and (2) providing techniques such as selection of optical flow algorithm, the pre-train dataset/architectures and the hyper-parameters for assembly in action recognition task. Experimental results show that we are able to obtain performance on a par with the state-of-the-art ones on the datasets of HMDB51 (70.9%) and UCF101 (95.4%).

DITIR

The prosperity of mobile social network and location-based services, e.g., Uber, is backing the explosive growth of spatial temporal streams on the Internet. It raises new challenges to the underlying data store system, which is supposed to support extremely high-throughput trajectory insertion and low-latency querying with spatial and temporal constraints. State-of-the-art solutions, e.g., HBase, do not render satisfactory performance, due to the high overhead on index update. In this demonstration, we present DITIR, our new system prototype tailored to efficiently processing temporal and spacial queries over historical data as well as latest updates. Our system provides better performance guarantee, by physically partitioning the incoming data tuples on their arrivals and exploiting a template-based insertion schema, to reach the desired ingestion throughput. Load balancing mechanism is also introduced to DITIR, by using which the system is capable of achieving reliable performance against workload dynamics. Our demonstration shows that DITIR supports over 1 million tuple insertions in a second, when running on a 10-node cluster. It also significantly outperforms HBase by 7 times on ingestion throughput and 5 times faster on query latency.

Miss it and miss out: Counterproductive nonspatial attentional capture by task-irrelevant, value-related stimuli.

Recent studies of visual search suggest that learning about valued outcomes (rewards and punishments) influences the likelihood that distractors will capture spatial attention and slow search for a target, even when those value-related distractors have never themselves been the targets of search. In the present study, we demonstrated a related effect in the context of temporal, rather than spatial, selection. Participants were presented with a temporal stream of pictures in a fixed central location and had to identify the orientation of a rotated target picture. Response accuracy was reduced if the rotated target was preceded by a "valued" distractor picture that signaled that a correct response to the target would be rewarded (and an incorrect response punished), relative to a distractor picture that did not signal reward or punishment. This effect of signal value on response accuracy was short-lived, being most prominent with a short lag between distractor and target. Impairment caused by a valued distractor was observed if participants were explicitly instructed regarding its relation to reward/punishment (Exps. 1, 3, and 4), or if they could learn this relationship only via trial-by-trial experience (Exp. 2). These findings show that the influence of signal value on attentional capture extends to temporal selection, and also demonstrate that value-related distractors can interfere with the conscious perception of subsequent target information.

Encoding of rapid time-varying information is impaired in poor readers.

A characteristic set of eye movements and fixations are made during reading, so the position of words on the retinae is constantly being updated. Effective decoding of print requires this temporal stream of visual information to be segmented or parsed into its constituent units (e.g., letters or words). Poor readers' difficulties with word recognition could arise at the point of segmenting time-varying visual information, but the mechanisms underlying this process are little understood. Here, we used random-dot displays to explore the effects of reading ability on temporal segmentation. Thirty-eight adult readers viewed test stimuli that were temporally segmented by constraining either local motions or analogous form cues to oscillate back and fourth at each of a range of rates. Participants had to discriminate these segmented patterns from comparison stimuli containing the same motion and form cues but these were temporally intermingled. Results showed that the motion and form tasks could not be performed reliably when segment duration was shorter than a temporal resolution (acuity) limit. The acuity limits for both tasks were significantly and negatively correlated with reading scores. Importantly, the minimum segment duration needed to detect the temporally segmented stimuli was longer in relatively poor readers than relatively good readers. This demonstrates that adult poor readers have difficulty segmenting temporally changing visual input particularly at short segment durations. These results are consistent with evidence suggesting that precise encoding of rapid time-varying information is impaired in developmental dyslexia.

Active demultiplexing of single photons from a solid‐state source

AbstractA scheme for active temporal‐to‐spatial demultiplexing of single photons generated by a solid‐state source is introduced. The scheme scales quasi‐polynomially with photon number, providing a viable technological path for routing n photons in the one temporal stream from a single emitter to n different spatial modes. Active demultiplexing is demonstrated using a state‐of‐the‐art photon source—a quantum‐dot deterministically coupled to a micropillar cavity—and a custom‐built demultiplexer—a network of electro‐optically reconfigurable waveguides monolithically integrated in a lithium niobate chip. The measured demultiplexer performance can enable a six‐photon rate three orders of magnitude higher than the equivalent heralded SPDC source, providing a platform for intermediate quantum computation protocols. image

Three-stream CNNs for action recognition

Existing Convolutional Neural Networks (CNNs) based methods for action recognition are either spatial or temporally local while actions are 3D signals. In this paper, we propose a global spatial-temporal three-stream CNNs architecture, which is able to be used for action feature extraction. Specifically, the three-stream CNNs comprises of spatial, local temporal and global temporal streams generated respectively from deep learning single frame, optical flow and global accumulated motion features in the form of a new formulation named Motion Stacked Difference Image (MSDI). Moreover, a novel soft Vector of Locally Aggregated Descriptors (soft-VLAD) is developed to further represent the extracted features, combining the advantage of Gaussian Mixture Models (GMMs) and VLAD by encoding data according to their overall probability distribution and the corresponding difference with respect to clustered centers. To deal with the inadequacy of training samples during learning, we introduce a data augmentation scheme which is very efficient due to its origin at cropping across videos. We conduct our experiments on UCF101 and HMDB51 datasets, and the results demonstrate the effectiveness of our approach.

Two-Stream Dictionary Learning Architecture for Action Recognition

In this paper, a novel method based on the two-stream dictionary learning architecture for human action recognition is proposed. The architecture consists of interest patch (IP) detector and descriptor, two-stream dictionary models, and support vector machine (SVM) for classification. The novel IP detector combines a human detector and a contour detector to extract patches of interest on human contours. Then the IP descriptors are calculated in spatial stream and temporal stream separately. In each stream, a dictionary is trained for each action with the IP descriptors as an action model. In this way, measuring the similarity between an action sequence and an action model is transformed to reconstructing the IPs in this sequence with the model and computing the reconstruction error. For each action, an IP distribution histogram is constructed and the histogram is further used to train an SVM classifier in each stream. A score fusion method is applied to fuse the spatial and temporal SVM classification results to make a final decision. The proposed architecture is examined on four public data sets with different background complexities and camera motion conditions: Weizmann data set, KTH data set, Olympic sports data set, and HMDB51 data set. The results are further compared with state-of-the-art approaches in the experiment section to confirm the effectiveness of this architecture.

The Empty Bower and the Lone Fountain

French poet-composer Guillaume de Machaut’s complete-works manuscripts are illuminated with hundreds of miniatures. Two manuscripts, made in his lifetime (c. 1300-1377), are among the most luxurious (BnF, ms. fr. 1584; BnF, ms. fr. 1586). I propose separating the iconographic programs from two dits (long, narrative poems cast in the first person) in these manuscripts to ‘analyze’ the resulting paratexts; perhaps anachronistic, but ultimately instructive in reconstructing period reception. This problematizes the role of the iconographer and his degree of responsibility in the creation of the images and/or their ‘arrangement’ within the text. Such an approach questions the concept of the supremacy of the text. Must an iconographic program only be judged by its rapport with the text? Can there be ‘right’ or ‘wrong’ ways in which the image is inserted into the text? Sixten Ringbom describes text-image bonds that are pertinent to understanding Machaut’s textual “Je” and the iconographer’s depiction of him. On occasion, the narrator disappears from the miniatures and the object of his gaze, unites the viewer, if only momentarily, with him – a static, visual ekphrasis. They constitute a form of ‘communication’ in and of themselves projected in a temporal stream of meaning. The concatenative sequence, hinging on discrete narratives, can be analyzed as a group. Subsequently the links can be removed, enabling an analysis of individual images. I will apply this methodology to the iconographic programs from Le dit dou vergier and La fonteinne amoureuse. The frontispiece for the Vergier depicts a large, intricately painted empty bower; the narrator is ancillary to this central image and stands off to one side. In the Fonteinne, two images of a fountain as subject matter, devoid of figures, break the narrative flow. This literally forces the viewer to focus on the subject matter rather than passively following the verbal action taking place around it. The goal of this study, then, is to reconsider the text as a means for interpreting the ‘correctness’ of the placement or content of the miniatures and to explore the inverse; c’est-a-dire, la theorisation de l’erreur.

High Accuracy Human Activity Recognition Based on Sparse Locality Preserving Projections.

Human activity recognition(HAR) from the temporal streams of sensory data has been applied to many fields, such as healthcare services, intelligent environments and cyber security. However, the classification accuracy of most existed methods is not enough in some applications, especially for healthcare services. In order to improving accuracy, it is necessary to develop a novel method which will take full account of the intrinsic sequential characteristics for time-series sensory data. Moreover, each human activity may has correlated feature relationship at different levels. Therefore, in this paper, we propose a three-stage continuous hidden Markov model (TSCHMM) approach to recognize human activities. The proposed method contains coarse, fine and accurate classification. The feature reduction is an important step in classification processing. In this paper, sparse locality preserving projections (SpLPP) is exploited to determine the optimal feature subsets for accurate classification of the stationary-activity data. It can extract more discriminative activities features from the sensor data compared with locality preserving projections. Furthermore, all of the gyro-based features are used for accurate classification of the moving data. Compared with other methods, our method uses significantly less number of features, and the over-all accuracy has been obviously improved.

CO2 time series patterns in contrasting headwater streams of North America

We explored the underlying patterns of temporal stream CO2 partial pressure (pCO2) variability using high-frequency sensors in seven disparate headwater streams distributed across the northern hemisphere. We also compared this dataset of >40,000 pCO2 records with other published records from lotic systems. Individual stream sites exhibited relatively distinct pCO2 patterns over time with few consistent traits across sites. Some sites showed strong diel variability, some exhibited increasing pCO2 with increasing discharge, whereas other streams had reduced pCO2 with increasing discharge or no clear response to changes in flow. The only “universal” signature observed in headwater streams was a late summer pCO2 maxima that was likely driven by greatest rates of organic matter respiration due to highest annual temperatures. However, we did not observe this seasonal pattern in a southern hardwood forest site, likely because the region was transitioning from a severe drought. This work clearly illustrates the heterogeneous nature of headwater streams, and highlights the idiosyncratic nature of a non-conservative solute that is jointly influenced by physics, hydrology, and biology. We suggest that future researchers carefully select sensor locations (within and among streams) and provide additional contextual information when attempting to explain pCO2 patterns.

Extracting Object Identity: Ventral or Dorsal Visual Stream?

The influential “two-streams hypothesis” of visual processing proposes that features related to object recognition are primarily encoded in the visual ventral–temporal stream (the “what” pathway), while spatial relationships among objects are primarily encoded in the dorsal–parietal

Mapping Temporal Variables Into the NeuCube for Improved Pattern Recognition, Predictive Modeling, and Understanding of Stream Data.

This paper proposes a new method for an optimized mapping of temporal variables, describing a temporal stream data, into the recently proposed NeuCube spiking neural network (SNN) architecture. This optimized mapping extends the use of the NeuCube, which was initially designed for spatiotemporal brain data, to work on arbitrary stream data and to achieve a better accuracy of temporal pattern recognition, a better and earlier event prediction, and a better understanding of complex temporal stream data through visualization of the NeuCube connectivity. The effect of the new mapping is demonstrated on three benchmark problems. The first one is the early prediction of patient sleep stage event from temporal physiological data. The second one is the pattern recognition of dynamic temporal patterns of traffic in the Bay Area of California and the last one is the Challenge 2012 contest data set. In all the cases, the use of the proposed mapping leads to an improved accuracy of pattern recognition and event prediction and a better understanding of the data when compared with traditional machine learning techniques or SNN reservoirs with an arbitrary mapping of the variables.