Temporal Pooling Research Articles

No-Reference Video Quality Assessment Based on Artifact Measurement and Statistical Analysis

A discrete cosine transform (DCT)-based no-reference video quality prediction model is proposed that measures artifacts and analyzes the statistics of compressed natural videos. The model has two stages: 1) distortion measurement and 2) nonlinear mapping. In the first stage, an unsigned ac band, three frequency bands, and two orientation bands are generated from the DCT coefficients of each decoded frame in a video sequence. Six efficient frame-level features are then extracted to quantify the distortion of natural scenes. In the second stage, each frame-level feature of all frames is transformed to a corresponding video-level feature via a temporal pooling, then a trained multilayer neural network takes all video-level features as inputs and outputs, a score as the predicted quality of the video sequence. The proposed method was tested on videos with various compression types, content, and resolution in four databases. We compared our model with a linear model, a support-vector-regression-based model, a state-of-the-art training-based model, and a four popular full-reference metrics. Detailed experimental results demonstrate that the results of the proposed method are highly correlated with the subjective assessments.

An efficient human visual system based quality metric for 3D video

Stereoscopic video technologies have been introduced to the consumer market in the past few years. A key factor in designing a 3D system is to understand how different visual cues and distortions affect the perceptual quality of stereoscopic video. The ultimate way to assess 3D video quality is through subjective tests. However, subjective evaluation is time consuming, expensive, and in some cases not possible. The other solution is developing objective quality metrics, which attempt to model the Human Visual System (HVS) in order to assess perceptual quality. Although several 2D quality metrics have been proposed for still images and videos, in the case of 3D efforts are only at the initial stages. In this paper, we propose a new full-reference quality metric for 3D content. Our method mimics HVS by fusing information of both the left and right views to construct the cyclopean view, as well as taking to account the sensitivity of HVS to contrast and the disparity of the views. In addition, a temporal pooling strategy is utilized to address the effect of temporal variations of the quality in the video. Performance evaluations showed that our 3D quality metric quantifies quality degradation caused by several representative types of distortions very accurately, with Pearson correlation coefficient of 90.8 %, a competitive performance compared to the state-of-the-art 3D quality metrics.

No-Reference Video Quality Assessment Model for Distortion Caused by Packet Loss in the Real-Time Mobile Video Services

Packet loss will make severe errors due to the corruption of related video data. For most video streams, because the predictive coding structures are employed, the transmission errors in one frame will not only cause decoding failure of itself at the receiver side, but also propagate to its subsequent frames along the motion prediction path, which will bring a significant degradation of end-to-end video quality. To quantify the effects of packet loss on video quality, a no-reference objective quality assessment model is presented in this paper. Considering the fact that the degradation of video quality significantly relies on the video content, the temporal complexity is estimated to reflect the varying characteristic of video content, using the macroblocks with different motion activities in each frame. Then, the quality of the frame affected by the reference frame loss, by error propagation, or by both of them is evaluated, respectively. Utilizing a two-level temporal pooling scheme, the video quality is finally obtained. Extensive experimental results show that the video quality estimated by the proposed method matches well with the subjective quality.

Implications of Time-Averaged Death Assemblages for Ecology and Conservation Biology

Biologists increasingly appreciate the importance of community-level attributes in the functioning and temporal turnover of ecosystems, but data other than species richness are difficult to acquire over the habitat-to-regional and decadal-to-millennial scales needed to recognize biodiversity change, discriminate between natural and anthropogenic drivers, and inform theoretical and applied ecology. Death assemblages (DAs)—the actively accumulating organic remains encountered in present-day seabeds and landscapes, as distinct from permanently buried fossil assemblages—are an underexploited source of historical information at precisely these scales. Meta-analyses, dynamic modeling, and individual case studies, particularly of mollusks and mammals, reveal that DAs differ from censused living assemblages (LAs) primarily because they are temporally coarse, time-averaged samples, contrary to concerns that postmortem bias dominates. Temporal pooling predictably damps the ability of DAs to detect small-scale variation, but promotes their ability to inventory rare species; estimate the abundance structure of the metacommunity; document range changes; evaluate historic habitat use; and identify now-absent species, community states, and anthropogenically shifted baselines.

Assessing quality of experience for high definition video streaming under diverse packet loss patterns

This paper presents an approach for deriving a QoE model of High Definition video streaming in the presence of different patterns of packet losses. The goal was achieved by using the SSIM video quality metric, temporal pooling techniques and content characteristics. Subjective tests were performed in order to verify the proposed models. The impact of several network loss patterns on diverse video content was analysed. The paper also deals with the encountered difficulties and presents intermediate steps to give a better understanding of the final result. The research aims to evaluate the perceived performance of IPTV and video surveillance systems. The model has been evaluated in the Quality of Experience (QoE) domain. The final model is generic and shows high correlation with the subjective results.

Content-Adaptive Packet-Layer Model for Quality Assessment of Networked Video Services

Packet-layer models are designed to use only the information provided by packet headers for real-time and non-intrusive quality monitoring of networked video services. This paper proposes a content-adaptive packet-layer (CAPL) model for networked video quality assessment. Considering the fact that the quality degradation of a networked video significantly relies on the temporal as well as the spatial characteristics of the video content, temporal complexity is incorporated in the proposed model. Due to very limited information directly available from packet headers, a simple and adaptive method for frame type detection is adopted in the CAPL model. The temporal complexity is estimated using the ratio of the number of bits for coding P and I frames. The estimated temporal complexity and frame type are incorporated in the CAPL model together with the information about the number of bits and positions of lost packets to obtain the quality estimate for each frame, by evaluating the distortions induced by both compression and packet loss. A two-level temporal pooling is employed to obtain the video quality given the frame quality. Using content related information, the proposed model is able to adapt to different video contents. Experimental results show that the CAPL model significantly outperforms the G.1070 model and the DT model in terms of widely used performance criteria, including the Root-Mean-Squared Error (RMSE), the Pearson Correlation Coefficient (PCC), the Spearman Rank Order Correlation Coefficient (SCC), and the Outlier Ratio (OR).

Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies

We introduce a new video quality database that models video distortions in heavily-trafficked wireless networks and that contains measurements of human subjective impressions of the quality of videos. The new LIVE Mobile Video Quality Assessment (VQA) database consists of 200 distorted videos created from 10 RAW HD reference videos, obtained using a RED ONE digital cinematographic camera. While the LIVE Mobile VQA database includes distortions that have been previously studied such as compression and wireless packet-loss, it also incorporates dynamically varying distortions that change as a function of time, such as frame-freezes and temporally varying compression rates. In this article, we describe the construction of the database and detail the human study that was performed on mobile phones and tablets in order to gauge the human perception of quality on mobile devices. The subjective study portion of the database includes both the differential mean opinion scores (DMOS) computed from the ratings that the subjects provided at the end of each video clip, as well as the continuous temporal scores that the subjects recorded as they viewed the video. The study involved over 50 subjects and resulted in 5,300 summary subjective scores and time-sampled subjective traces of quality. In the behavioral portion of the article we analyze human opinion using statistical techniques, and also study a variety of models of temporal pooling that may reflect strategies that the subjects used to make the final decision on video quality. Further, we compare the quality ratings obtained from the tablet and the mobile phone studies in order to study the impact of these different display modes on quality. We also evaluate several objective image and video quality assessment (IQA/VQA) algorithms with regards to their efficacy in predicting visual quality. A detailed correlation analysis and statistical hypothesis testing is carried out. Our general conclusion is that existing VQA algorithms are not well-equipped to handle distortions that vary over time. The LIVE Mobile VQA database, along with the subject DMOS and the continuous temporal scores is being made available to researchers in the field of VQA at no cost in order to further research in the area of video quality assessment.

Video Quality Pooling Adaptive to Perceptual Distortion Severity

It is generally recognized that severe video distortions that are transient in space and/or time have a large effect on overall perceived video quality. In order to understand this phenomena, we study the distribution of spatio-temporally local quality scores obtained from several video quality assessment (VQA) algorithms on videos suffering from compression and lossy transmission over communication channels. We propose a content adaptive spatial and temporal pooling strategy based on the observed distribution. Our method adaptively emphasizes "worst" scores along both the spatial and temporal dimensions of a video sequence and also considers the perceptual effect of large-area cohesive motion flow such as egomotion. We demonstrate the efficacy of the method by testing it using three different VQA algorithms on the LIVE Video Quality database and the EPFL-PoliMI video quality database.

13C-dating, the first method to calculate the relative age of molecular substance homologues in soil

This article reports the design of 13C-dating, the first method to calculate the relative age of molecular substance homologues occurring in fractions from the same soil sample. Soil is a major carbon pool impacting modern climate by CO2 release and uptake. Molecular substances that sequester carbon in soils are poorly known due to the absence of methods to study molecular-level C dynamics over agricultural time scales, e.g., 0–200 years. Here, I design a method to calculate the relative age of the plant-derived C31n-alkane occurring in 6 fractions from a soil sample naturally 13C-labelled by maize cropping during 23 years. Soil fractions are the bulk soil extract, two humin-encapsulated fractions and three particle-size fractions. Results show that C31n-alkane homologues have relative ages ranging from −6.7 years for the humin-encapsulated homologue to +25.1 years for the 200–2,000-μm fraction homologue. Such a wide variation of 31.8 years evidences temporal pools of molecular substances in soil. This finding also reveals that physical encapsulation can strikingly change the dynamics of a single molecular substance. 13C-dating thus allows to assess the carbon storage potential of molecular substances from crop soils. Such knowledge will help to identify molecular compounds, associated soil pools and agricultural practices that favour carbon sequestration. 13C-dating is further applicable to any environmental sample containing organic matter subjected to a 13C isotope shift with time. 13C-dating will also help to study the sequestration and delayed release of chemicals in various disciplines, such as pollutants in environmental sciences, pharmaceuticals in medicine, and nutrients in food science.

Real‐time quality assessment for voice over IP

SUMMARYA nonintrusive packet layer model is proposed for quality assessment of Voice over Internet Protocol. Through a simple analysis on the packet headers of the bit‐stream, the information about the bit‐rate and lost status is obtained for each frame. Then the frame quality is measured in terms of coding distortion, frame loss, and contamination because of error propagation. Given the frame quality, a two‐level temporal pooling scheme is designed to predict the overall speech quality. Experimental results show that the proposed model is very accurate in quality assessment. Moreover, the proposed packet layer model is featured by its low complexity, which makes it very suitable for real‐time quality monitoring for Voice over Internet Protocol services. Copyright © 2011 John Wiley & Sons, Ltd.

SPATIAL-TEMPORAL VIDEO QUALITY ASSESSMENT BASED ON TWO-LEVEL TEMPORAL POOLING

In this paper, a video quality assessment method based on two-level temporal pooling is proposed. By dividing a video sequence into groups of frames (GOFs) with variable lengths, a short-term temporal pooling is performed first at the eye fixation level to obtain the GOF quality. The determination of the GOF size is based on empirical observations from subjective tests that evaluate the duration of successive frames based on which steady quality judgment can be made by the human visual system. The video quality is further obtained by a second level of long-term temporal pooling using the GOF quality. The impact of strong impairment in part of the video on the cognitive system is incorporated in the proposed method for video quality assessment. Extensive experimental results have demonstrated the effectiveness of the proposed method for spatial-temporal distortion evaluation considering both regular and irregular frame loss.

A Computational Intelligibility Model for Assessment and Compression of American Sign Language Video

Real-time, two-way transmission of American Sign Language (ASL) video over cellular networks provides natural communication among members of the Deaf community. As a communication tool, compressed ASL video must be evaluated according to the intelligibility of the conversation, not according to conventional definitions of video quality. Guided by linguistic principles and human perception of ASL, this paper proposes a full-reference computational model of intelligibility for ASL (CIM-ASL) that is suitable for evaluating compressed ASL video. The CIM-ASL measures distortions only in regions relevant for ASL communication, using spatial and temporal pooling mechanisms that vary the contribution of distortions according to their relative impact on the intelligibility of the compressed video. The model is trained and evaluating using ground truth experimental data collected in three separate studies. The CIM-ASL provides accurate estimates of subjective intelligibility and demonstrates statistically significant improvements over computational models traditionally used to estimate video quality. The CIM-ASL is incorporated into an H.264 compliant video coding framework, creating a closed-loop encoding system optimized explicitly for ASL intelligibility. The ASL-optimized encoder achieves bitrate reductions between 10% and 42%, without reducing intelligibility, when compared to a general purpose H.264 encoder.

Relating spatial and temporal orientation pooling to population decoding solutions in human vision

Spatial pooling is often considered synonymous with averaging (or other statistical combinations) of local information contained within a complex visual image. We have recently shown, however, that spatial pooling of motion signals is better characterized in terms of optimal decoding of neuronal populations rather than image statistics (Webb et al., 2007). Here we ask which computations guide the spatial and temporal pooling of local orientation signals in human vision. The observers’ task was to discriminate which of two texture patterns had a more clockwise global orientation. Standard textures had a common orientation; comparison textures were chosen independently from a skewed (asymmetrical) probability distribution with distinct spatial or temporal statistics. We simulated observers’ performance using different estimators (vector average, winner-takes-all and maximum likelihood) to decode the orientation-tuned activity of a population of model neurons. Our results revealed that the perceived global orientation of texture patterns coincided with the mean (or vector average read-out) of orientation signals accumulated over both space and time. To reconcile these results with our previous work on direction pooling, we varied stimulus duration. Perceived global orientation was accurately predicted by a vector average read-out of orientation signals at relatively short stimulus durations and maximum likelihood read-out at longer durations. Moreover, decreasing the luminance contrast of texture patterns increased the duration of the transition from a vector average to maximum likelihood read-out. Our results suggest that direction and orientation pooling use similar probabilistic read-out strategies when sufficient time is available.

Seasonal Occurrence of Immature Mosquitoes in Swimming Pools in Buenos Aires, Argentina

The seasonal variation in the density of swimming pools and the occurrence of immature mosquitoes in these habitats were studied in the temperate city of Buenos Aires, Argentina. The density of swimming pools was highest in the summer (6.0 pools/ha) when temporal pools predominated; it was lowest in the winter (0.9 pools/ha) when only permanent pools were recorded. The presence of immature stages of mosquitoes was mainly associated with the lack of use and maintenance of the pools in the fall season, when temporal pools were still assembled but not intensively used. Seven mosquito species were recorded, among which Culex pipiens and Culex apicinus were the most frequent.

Considering Temporal Variations of Spatial Visual Distortions in Video Quality Assessment

The temporal distortions such as flickering, jerkiness, and mosquito noise play a fundamental part in video quality assessment. A temporal distortion is commonly defined as the temporal evolution, or fluctuation, of the spatial distortion on a particular area which corresponds to the image of a specific object in the scene. Perception of spatial distortions over time can be largely modified by their temporal changes, such as increase or decrease in the distortions, or as periodic changes in the distortions. In this paper, we have designed a perceptual full reference video quality assessment metric by focusing on the temporal evolutions of the spatial distortions. As the perception of the temporal distortions is closely linked to the visual attention mechanisms, we have chosen to first evaluate the temporal distortion at eye fixation level. In this short-term temporal pooling, the video sequence is divided into spatio-temporal segments in which the spatio-temporal distortions are evaluated, resulting in spatio-temporal distortion maps. Afterwards, the global quality score of the whole video sequence is obtained by the long-term temporal pooling in which the spatio-temporal maps are spatially and temporally pooled. Consistent improvement over objective existing video quality assessment methods is observed. Our validation has been realized with a dataset built from video sequences of various contents.

Motion processing at low light levels: Differential effects on the perception of specific motion types

While many aspects of human vision at low light levels have been studied in great detail, motion perception has rarely been investigated so far. Here we address differential effects of light level on the perception of coherent motion, heading from radial flow, and biological motion. We determined detection thresholds under photopic, mesopic, and scotopic conditions. Results indicate that the perception of specific motion types differs in vulnerability to changes in light level. Thresholds for coherent motion and heading from radial flow increased monotonically from photopic to mesopic and scotopic light levels. We suppose that observed deficits are due to temporal pooling under rod-dominated vision. In contrast, detection thresholds for biological motion, which is distinguished by temporal dynamics and a specific spatial distribution of nearby signals, were exclusively elevated under mesopic conditions. Thresholds under scotopic conditions matched those under photopic conditions. Selective constraints under mesopic conditions might be explained by a detrimental interaction of rod and cone vision as well as by activity of different rod pathways. Findings suggest that very early retinal signal processing can have complex effects on the perception of different motion types, which is generally considered to rely on cortical areas.

Vision in the nocturnal wandering spider Leucorchestris arenicola (Araneae: Sparassidae)

At night the Namib Desert spider Leucorchestris arenicola performs long-distance homing across its sand dune habitat. By disabling all or pairs of the spiders' eight eyes we found that homing ability was severely reduced when vision was fully abolished. Vision, therefore, seems to play a key role in the nocturnal navigational performances of L. arenicola. After excluding two or three pairs of eyes, the spiders were found to be able to navigate successfully using only their lateral eyes or only their anterior median eyes. Measurement of the eyes' visual fields showed that the secondary eyes combined have a near full (panoramic) view of the surroundings. The visual fields of the principal eyes overlap almost completely with those of the anterior lateral eyes. Electroretinogram recordings indicate that each eye type contains a single photopigment with sensitivity peaking at approximately 525 nm in the posterior and anteriomedian eyes, and at approximately 540 nm in the anteriolateral eyes. Theoretical calculations of photon catches showed that the eyes are likely to employ a combination of spatial and temporal pooling in order to function at night. Under starlit conditions, the raw spatial and temporal resolution of the eyes is insufficient for detecting any visual information on structures in the landscape, and bright stars would be the only objects visible to the spiders. However, by summation in space and time, the spiders can rescue enough vision to detect coarse landscape structures. We show that L. arenicola spiders are likely to be using temporal summation to navigate at night.

Kinetics of Fatty Acid Biohydrogenation In Vitro

Biohydrogenation (BH) of fatty acids (FA) from fresh alfalfa and alfalfa hay with varying levels of supplemental sucrose and media pH was evaluated in vitro. A multicompartmental model was then developed to estimate pool size and flux of vaccenic acid (VA) during BH of FA in fresh alfalfa. To vary incubation pH, alfalfa samples were inoculated with rumen fluid in 2 media differing in molarity of the bicarbonate buffer. Samples were incubated for 0, 1, 2, 3, 4, 6, 9, and 12h; pH was measured and tubes were put in ice and stored until analysis. The BH rates of linoleic acid (18:2) and linolenic acid (18:3) were estimated by PROC NLIN of SAS (single pool, first-order kinetic model) and SAAM II (multiple pools, first-order kinetic model). Both methods gave similar estimates for the BH rates of 18:2 and 18:3 as well as the temporal pool size of VA. The BH rates (%/h) in the strong (SB) and weak buffers (WB) were 27.4 (±0.7) and 23.5 (±0.9) for 18:2, and 43.8 (±0.2) and 30.3 (±0.6) for 18:3, respectively. The WB decreased the BH rates of 18:2 and 18:3 for both forage sources. However, BH rates of 18:3 were higher from fresh alfalfa than alfalfa hay. There was no effect of sucrose addition on the BH rates of 18:2 and 18:3. Moreover, there was no effect of buffer on the BH of VA estimated by the multiple pools model between the SB and WB (12.5±2.1 and 14.1±3.7%/h, respectively). The BH rates of the conjugated linoleic acid isomers were not different between the SB and WB treatments (36.7±19.8 and 25.9±27.2, respectively). Because we could estimate fluxes as well as mass of the VA pools, more information was generated from the data when a multiple pools model was used compared with a single pool, first-order kinetic model.

A Convolutional Neural Network Approach for Objective Video Quality Assessment

This paper describes an application of neural networks in the field of objective measurement method designed to automatically assess the perceived quality of digital videos. This challenging issue aims to emulate human judgment and to replace very complex and time consuming subjective quality assessment. Several metrics have been proposed in literature to tackle this issue. They are based on a general framework that combines different stages, each of them addressing complex problems. The ambition of this paper is not to present a global perfect quality metric but rather to focus on an original way to use neural networks in such a framework in the context of reduced reference (RR) quality metric. Especially, we point out the interest of such a tool for combining features and pooling them in order to compute quality scores. The proposed approach solves some problems inherent to objective metrics that should predict subjective quality score obtained using the single stimulus continuous quality evaluation (SSCQE) method. This latter has been adopted by video quality expert group (VQEG) in its recently finalized reduced referenced and no reference (RRNR-TV) test plan. The originality of such approach compared to previous attempts to use neural networks for quality assessment, relies on the use of a convolutional neural network (CNN) that allows a continuous time scoring of the video. Objective features are extracted on a frame-by-frame basis on both the reference and the distorted sequences; they are derived from a perceptual-based representation and integrated along the temporal axis using a time-delay neural network (TDNN). Experiments conducted on different MPEG-2 videos, with bit rates ranging 2-6 Mb/s, show the effectiveness of the proposed approach to get a plausible model of temporal pooling from the human vision system (HVS) point of view. More specifically, a linear correlation criteria, between objective and subjective scoring, up to 0.92 has been obtained on a set of typical TV videos.

Visualizing Live Text Streams Using Motion and Temporal Pooling

The informational context in which we live and work is becoming increasingly complex and rich, with information much more plentiful and accessible than ever before. Our informational context is also faster paced, with news and developments spreading across media and the Internet at an often astounding rate. In today's competitive world, the first response to a trend is usually the most effective. Thus far, the technology responsible for this information complexity offers little to help us quickly understand and act upon it. Although visualization is useful, its techniques are typically too slow or simply not designed to help users understand highly time-sensitive and dynamic information streams. To address this, we developed TextPool, a system that uses temporal pooling to visualize live text streams such as newswires and system telemetry. In temporal pooling, systems gather recent stream content into a buffer (pool) to use for visualization. Because the stream continues to flow through the pool in real time, the visualization must be dynamic, using motion to respond to the pool's changing content. The TextPool system processes live text streams in real time using information retrieval techniques, extracts the most frequently occurring salient terms from the buffered streams, and displays related terms next to one another in a dynamic text collage.