Temporal Masking Research Articles

Satisfied-User-Ratio Modeling for Compressed Video.

With explosive increase of internet video services, perceptual modeling for video quality has attracted more attentions to provide high quality-of-experience (QoE) for end-users subject to bandwidth constraints, especially for compressed video quality. In this paper, a novel perceptual model for satisfied-user-ratio (SUR) on compressed video quality is proposed by exploiting compressed video bitrate changes and spatial-temporal statistical characteristics extracted from both uncompressed original video and reference video. In the proposed method, an efficient video feature set is explored and established to model SUR curves against bitrate variations by leveraging the Gaussian Processes Regression (GPR) framework. In particular, the proposed model is based on the recently released large-scale video quality dataset, VideoSet, and takes both spatial and temporal masking effects into consideration. To make it more practical, we further optimize the proposed method from three aspects including feature source simplification, computation complexity reduction and video codec adaption. Based on experimental results on VideoSet, the proposed method can accurately model SUR curves for various video contents and predict their required bitrates at given SUR values. Subjective experiments are conducted to further verify the generalization ability of the proposed SUR model.

Color Video JND Model Using Compound Spatial Masking and Structure-Based Temporal Masking

In order to provide high transmission efficiency and visual quality for the color image/video in internet, effective estimation of its inherent just noticeable distortion (JND) is continuously an important topic. In this paper, a discrete cosine transform (DCT)-based perceptual model for estimating spatio-temporal JNDs of color videos is presented. Firstly, the compound masking adjustment integrated with different spatial masking factors is proposed. Based on the base detection threshold for DCT coefficients in luminance and chrominance components of the color image, the adjustment combined by luminance masking, pattern-based contrast masking, and cross masking is exploited to measure visibility thresholds for luminance component, while the adjustment induced by the variance-based statistical properties is utilized to measure visibility thresholds for chrominance components. Then, the local temporal statistics of luminance component are considered to design the structure-based temporal masking adjustment for further estimating visibility thresholds of video signals. To verify the proposed method, a subjective viewing test is designed and a fair rating process of evaluating the visual quality is carried out under the specified viewing condition. Experimental results show that the proposed method is able to measure more visual distortion tolerance than the existing color-based methods at high visual quality.

Robust Data-Driven Auditory Profiling Towards Precision Audiology.

The sources and consequences of a sensorineural hearing loss are diverse. While several approaches have aimed at disentangling the physiological and perceptual consequences of different etiologies, hearing deficit characterization and rehabilitation have been dominated by the results from pure-tone audiometry. Here, we present a novel approach based on data-driven profiling of perceptual auditory deficits that attempts to represent auditory phenomena that are usually hidden by, or entangled with, audibility loss. We hypothesize that the hearing deficits of a given listener, both at hearing threshold and at suprathreshold sound levels, result from two independent types of “auditory distortions.” In this two-dimensional space, four distinct “auditory profiles” can be identified. To test this hypothesis, we gathered a data set consisting of a heterogeneous group of listeners that were evaluated using measures of speech intelligibility, loudness perception, binaural processing abilities, and spectrotemporal resolution. The subsequent analysis revealed that distortion type-I was associated with elevated hearing thresholds at high frequencies and reduced temporal masking release and was significantly correlated with elevated speech reception thresholds in noise. Distortion type-II was associated with low-frequency hearing loss and abnormally steep loudness functions. The auditory profiles represent four robust subpopulations of hearing-impaired listeners that exhibit different degrees of perceptual distortions. The four auditory profiles may provide a valuable basis for improved hearing rehabilitation, for example, through profile-based hearing-aid fitting.

Differences and similarities between temporal crowding, spatial crowding and masking

Differences and similarities between temporal crowding, spatial crowding and masking

Second language vocabulary level is related to benefits for second language listening comprehension under lower reverberation time conditions

ABSTRACTThe acoustic qualities of a room can have a deleterious effect on the quality of speech signals. The acoustic measurement of reverberation time (RT) has shown to impact second language (L2) speech comprehension positively when lower due to release from spectral and temporal masking effects as well as top-down processing factors. This auralization experiment investigated the benefits of better L2 vocabulary and executive function (updating) skills during L2 listening comprehension tests under shorter versus longer RT conditions (0.3 and 0.9 s). 57 bilingual university students undertook L2 vocabulary, number updating and L2 listening comprehension tests. After splitting groups into high/low vocabulary and updating groups, a mixed ANOVA was conducted. The high number updating group showed no significant differences or interactions in L2 listening comprehension than the lower number updating group across RT conditions. The high vocabulary group had 22% better L2 listening comprehension than the low vocabulary group in long RT, and 9% better in short RT. A significant benefit in L2 listening comprehension due to release from reverberation was only evident in the high vocabulary group. Results indicate that the benefit of good room acoustics for listening comprehension is greatest for those with better language (vocabulary) ability.

Forward masking enhances the auditory brainstem response in the free-tailed bat, Tadarida brasiliensis, during a critical time window for sonar reception.

Forward masking is a widespread auditory phenomenon in which the response to one sound transiently reduces the response to a succeeding sound. This study used auditory brainstem responses to measure temporal masking effects in the free-tailed bat, Tadarida brasiliensis. A digital subtraction protocol was used to isolate responses to the second of a pair of pulses varying in interval, revealing a suppression phase lasting <4 ms followed by an enhancement phase lasting 4-15 ms during which the ABR waveform was amplified up to 100%. The results suggest echolocating bats possess adaptations for enhancing sonar receiver gain shortly after pulse emission.

Time and magnitude dependent masking effects of vehicle's 2 Hz shock type vibrations on the perception of vehicle's 4.5 Hz longitudinal shock type vibrations for seated human subjects

In the literature there are no known studies regarding temporal masking effects of a vehicle's shock type vibrations. Since shock type vibrations of different directions of movement, magnitudes and frequencies are acting on the human body while driving a vehicle, temporal masking effects may be relevant when evaluating a vehicle's ride comfort. This study investigates temporal post-masking effects of a vehicle's 2 Hz shock type vibrations (masker) on the perception of 4.5 Hz longitudinal shock type vibrations (test signal) for the seated human body. It was found that post-masking exists for the investigated vibrations. The masking effect is highly dependent on the time gap between masker and test signal: with increasing time gap the masking effect decreases. Furthermore the amount of masking increases with increasing masker magnitude. Both findings are consistent with the findings in psychoacoustics. In addition to that the masking effect depends on the masker's direction of movement.

Time and magnitude dependent masking effects of vehicle's 2 Hz shock type vibrations on the perception of vehicle's 4.5 Hz longitudinal shock type vibrations for seated human subjects

In the literature there are no known studies regarding temporal masking effects of a vehicle's shock type vibrations. Since shock type vibrations of different directions of movement, magnitudes and frequencies are acting on the human body while driving a vehicle, temporal masking effects may be relevant when evaluating a vehicle's ride comfort. This study investigates temporal post-masking effects of a vehicle's 2 Hz shock type vibrations (masker) on the perception of 4.5 Hz longitudinal shock type vibrations (test signal) for the seated human body. It was found that post-masking exists for the investigated vibrations. The masking effect is highly dependent on the time gap between masker and test signal: with increasing time gap the masking effect decreases. Furthermore the amount of masking increases with increasing masker magnitude. Both findings are consistent with the findings in psychoacoustics. In addition to that the masking effect depends on the masker's direction of movement.

A Just Noticeable Difference-Based Video Quality Assessment Method with Low Computational Complexity

A Just Noticeable Difference (JND)-based video quality assessment (VQA) method is proposed. This method, termed as JVQ, applies JND concept to structural similarity (SSIM) index to measure the spatial quality. JVQ incorporates three features, i.e. luminance adaptation, contrast masking, and texture masking. In JVQ, the concept of JND is refined and more features are considered. For the spatial part, minor distortions in the distorted frames are ignored and considered imperceptible. For the temporal part, SSIM index is simplified and used to measure the temporal video quality. Then, a similar JND concept which comprises of temporal masking is also applied in the temporal quality evaluation. Pixels with large variation over time are considered as not distorted because the distortions in these pixels are hardly perceivable. The final JVQ index is the arithmetic mean of both spatial and temporal quality indices. JVQ is found to achieve good correlation with subjective scores. In addition, this method has low computational cost as compared to existing state-of-the-art metrics.

Analysis of Temporal Masking Effects on Master- and Slave-Type Flip-Flop SEUs and Related Applications

Analysis of temporal masking effects of single-event upsets (SEUs) on master–slave flip-flops (FFs) is provided, considering the difference of SEU cross sections between master and slave latches. An improved model of temporal masking effects of SEUs in master–slave FFs is proposed and demonstrated through alpha-particle single-event effect (SEE) experiments. Based on the improved model, an effective and accurate methodology to quantitatively evaluate single-event transient and SEU-induced error cross sections of FF chains is proposed. Implications of SEE hardening for sequential circuits are discussed.

Video quality assessment accounting for temporal visual masking of local flicker

An important element of the design of video quality assessment (VQA) models that remains poorly understood is the effect of temporal visual masking on the visibility of temporal distortions. The visibility of temporal distortions like local flicker can be strongly reduced by motion. Based on a recently discovered visual change silencing illusion, we have developed a full reference VQA model that accounts for temporal visual masking of local flicker. The proposed model, called Flicker Sensitive-MOtion-based Video Integrity Evaluation (FS-MOVIE), augments the well-known MOVIE Index by combining motion tuned video integrity features with a new perceptual flicker visibility/masking index. FS-MOVIE captures the separated spectral signatures caused by local flicker distortions, by using a model of the responses of neurons in primary visual cortex to video flicker, an energy model of motion perception, and a divisive normalization stage. FS-MOVIE predicts the perceptual suppression of local flicker by the presence of motion and evaluates local flicker as it affects video quality. Experimental results show that FS-MOVIE significantly improves VQA performance against its predecessor and is highly competitive with top performing VQA algorithms when tested on the LIVE, IVP, EPFL, and VQEGHD5 VQA databases.

Correlations between otoacoustic emissions and performance in common psychoacoustical tasks.

Performance was measured on seven common psychoacoustical tasks for about 75 highly trained subjects. Because some psychoacoustical outcomes varied by race, the subjects were partitioned into White and Non-White categories for analysis. Sex, race, and menstrual-cycle differences in performance are described in a companion paper [McFadden, Pasanen, Maloney, Leshikar, and Pho (2018). J. Acoust. Soc. Am. 143, 2338-2354]. Also measured for all subjects were three types of otoacoustic emissions (OAEs): spontaneous otoacoustic emissions (SOAEs), click-evoked otoacoustic emissions (CEOAEs), and distortion-product otoacoustic emissions (DPOAEs). The experimental question was whether and how OAEs were correlated with psychoacoustical performance. In accord with past findings, the SOAEs and CEOAEs exhibited substantial sex and race differences, but the DPOAEs did not. Somewhat surprisingly, the correlations between OAEs and psychoacoustical performance were generally weak. No form of OAE was highly correlated with any psychoacoustical task for both sexes within a race category. Thus, there was no compelling evidence that the mechanisms underlying OAEs also contribute systematically to performance in any of the simultaneous or temporal masking tasks studied here. Especially surprising were the weak correlations between OAEs and detection of a tone in the quiet. Apparently individual differences in psychoacoustical performance reside more in post-cochlear (neural) mechanisms than in individual differences in the cochlear ("mechanical") mechanisms underlying the OAEs measured here.

Can monaural temporal masking explain the ongoing precedence effect?

The precedence effect for transient sounds has been proposed to be based primarily on monaural processes, manifested by asymmetric temporal masking. This study explored the potential for monaural explanations with longer ("ongoing") sounds exhibiting the precedence effect. Transient stimuli were single lead-lag noise burst pairs; ongoing stimuli were trains of 63 burst pairs. Unlike with transients, monaural masking data for ongoing sounds showed no advantage for the lead, and are inconsistent with asymmetric audibility as an explanation for ongoing precedence. This result, along with supplementary measurements of interaural time discrimination, suggests different explanations for transient and ongoing precedence.

Formal Methods Based Synthesis of Single Event Transient Tolerant Combinational Circuits

Recent radiation ground testing campaigns of digital designs have demonstrated that the probability for Single Event Transient (SET) propagation is increasing in advanced technologies. This paper presents a hierarchical reliability-aware synthesis framework to design combinational circuits at gate level with minimal area overhead. This framework starts by estimating the vulnerability of the circuit to SETs. This is done by modeling the SET propagation as a Satisfiability problem by utilizing Satisfiability Modulo Theories (SMTs). An all-solution SMT solver is adapted to estimate the soft error rate due to SETs. Different strategies to mitigate SETs are integrated in the proposed framework to selectively harden vulnerable nodes in the design. Both logical and temporal masking factors of the target circuit are improved to harden sensitive paths or sub-circuits, whose SET propagation probability is relatively high. This process is repeated until the desired soft error rate is achieved or a given area overhead constraint is met. The proposed framework was implemented on different combinational designs. The reliability of a circuit can be improved by 64% with less than 20% area overhead.

The Tight-interlocked Rhythm Section: Production and Perception of Synchronisation in Jazz Trio Performance

This study investigates the production and perception of timing, synchronisation and dynamics in jazz trio performances. In a production experiment, six trio combinations of one saxophonist, two bassists, and three drummers were recorded while they performed three popular jazz songs. Onset timing and dynamics of each performer were extracted and analysed. Results showed that the tempo was significantly influenced by the timing of the drummers and all performers showed higher temporal precision on the backbeats. The drummers demonstrated individual swing-ratios, accentuations of beats and intrapersonal asynchronies between simultaneous hi-hat and ride cymbal onsets, which resulted in a hi-hat played 2–26 ms ahead of the pulse of the music. In a subsequent perception test, participants ( ) rated 12 excerpts of the jazz recordings. They selected their preferred version from a pool of stimuli containing the original version, but also manipulations with artificially increased or reduced asynchronies. Stimuli with reduced asynchronies smaller than 19 ms were preferred by the listeners over the original or the fully quantised timing. This suggests that listeners endorse a ‘tight-interlocked’ jazz rhythm section, with asynchronies smaller than the perceptual threshold (temporal masking), but with natural timing variabilities that makes it distinguishable from a computer-generated playback.

Saccadic suppression during voluntary versus reactive saccades.

Saccades are fast eye movements that reorient gaze. They can be performed voluntarily-for example, when viewing a scene-but they can also be triggered in reaction to suddenly appearing targets. The generation of these voluntary and reactive saccades have been shown to involve partially different cortical pathways. However, saccades of either type confront the visual system with a major challenge from massive image motion on the retina. Despite the fact that the whole scene is swept across the retina, a saccade usually does not elicit a percept of motion. This saccadic omission has been linked to a transient decrease of visual sensitivity during the eye movement, a phenomenon called saccadic suppression. A passive origin of saccadic suppression based on temporal masking has been proposed as well as an active central process that inhibits visual processing during the saccade. The latter one would need to include an extraretinal signal, which is generated already during saccade preparation. Since saccade generation differs for voluntary and reactive saccades, timing and nature of this extraretinal signal as well as its impact on visual sensitivity might also differ. We measured detection thresholds for luminance stimuli that were flashed during voluntary and reactive saccades and during fixation. Detection thresholds were higher during voluntary than during reactive saccades such that suppression appeared stronger during voluntary saccades. Stronger suppression in voluntary saccades could arise from a stronger extraretinal signal that activates suppression or could indicate that a suppression underlying process itself partially differs between voluntary and reactive saccades.

A DCT-Based Total JND Profile for Spatiotemporal and Foveated Masking Effects

In image and video processing fields, Discrete Cosine Transform (DCT)-based just-noticeable difference (JND) profiles have effectively been utilized to remove perceptual redundancies in pictures for compression. In this paper, we solve two problems that are often intrinsic to the conventional DCT-based JND profiles: 1) no foveated masking (FM) JND model has been incorporated in modeling the DCT-based JND profiles and 2) the conventional temporal masking (TM) JND models assume that all moving objects in frames can be well tracked by the eyes and that they are projected on the fovea regions of the eyes, which is not a realistic assumption and may result in poor estimation of JND values for untracked moving objects (or image regions). To solve these two problems, we first propose a generalized JND model for joint effects between TM and FM effects. With this model, called the temporal–foveated masking (TFM) JND model, JND thresholds for any tracked/untracked and moving/still image regions can be elaborately estimated. Finally, the TFM-JND model is incorporated into a total DCT-based JND profile with a spatial contrast sensitivity function, luminance masking, and contrast masking JND models. In addition, we propose a JND adjustment method for our total JND profile to avoid overestimation of JND values for image blocks of fixed sizes with various image characteristics. To validate the effectiveness of the total JND profile, an experiment involving a subjective distortion-visibility assessment has been conducted. The experiment results show that the proposed total DCT-based JND profile yields significant performance improvement with much higher capability of distortion concealment (average 5.6-dB lower PSNR) compared with state-of-the-art JND profiles. The MATLAB source code of the proposed total DCT-based JND profile is publicly available online at https://sites.google.com/site/sunghobaecv/jnd .

On the relationships between otoacoustic emissions, auditory evoked potentials, and psychoacoustical performance

Performance was measured on several common psychoacoustical tasks for about 70 subjects. The measures included simultaneous and temporal masking, masking by tones and by complex sounds, critical bandwidth, release from masking, and detection in the quiet. Also measured were spontaneous, click-evoked, and distortion-product otoacoustic emissions (OAEs) and auditory evoked potentials (AEPs, short and middle latency). Of interest were the correlations between psychoacoustical performance and the various physiological measures as well as any mean differences by sex and by menstrual cycle. Subjects were tested behaviorally in same-sex crews of 4–8 members, and behavioral testing required from 8 to 10 weeks for each crew. Correlation and effect size were the primary measures of interest. Resampling was used to determine implied significance for the various comparisons studied. Some correlations between physiological measures were moderately high, but the correlations between psychoacoustical tasks and the different physiological measures were generally low, although there were some unexpected differences by racial background. That is, the individual differences observed in psychoacoustical performance generally were not related to the individual differences in the various physiological measures. For these subjects, at least, psychoacoustical performance seemed unrelated to the mechanisms underlying OAEs and AEPs. [Work supported by NIDCD (DC000153).]

Revealing information by averaging.

We present a method for hiding images in synthetic videos and reveal them by temporal averaging. The main challenge is to develop a visual masking method that hides the input image both spatially and temporally. Our masking approach consists of temporal and spatial pixel by pixel variations of the frequency band coefficients representing the image to be hidden. These variations ensure that the target image remains invisible both in the spatial and the temporal domains. In addition, by applying a temporal masking function derived from a dither matrix, we allow the video to carry a visible message that is different from the hidden image. The image hidden in the video can be revealed by software averaging, or with a camera, by long-exposure photography. The presented work may find applications in the secure transmission of digital information.

Investigating time-efficiency of forward masking paradigms for estimating basilar membrane input-output characteristics.

It is well known that pure-tone audiometry does not sufficiently describe individual hearing loss (HL) and that additional measures beyond pure-tone sensitivity might improve the diagnostics of hearing deficits. Specifically, forward masking experiments to estimate basilar-membrane (BM) input-output (I/O) function have been proposed. However, such measures are very time consuming. The present study investigated possible modifications of the temporal masking curve (TMC) paradigm to improve time and measurement efficiency. In experiment 1, estimates of knee point (KP) and compression ratio (CR) of individual BM I/Os were derived without considering the corresponding individual “off-frequency” TMC. While accurate estimation of KPs was possible, it is difficult to ensure that the tested dynamic range is sufficient. Therefore, in experiment 2, a TMC-based paradigm, referred to as the “gap method”, was tested. In contrast to the standard TMC paradigm, the maker level was kept fixed and the “gap threshold” was obtained, such that the masker just masks a low-level (12 dB sensation level) signal. It is argued that this modification allows for better control of the tested stimulus level range, which appears to be the main drawback of the conventional TMC method. The results from the present study were consistent with the literature when estimating KP levels, but showed some limitations regarding the estimation of the CR values. Perspectives and limitations of both approaches are discussed.