Audio-visual Synchronization Research Articles

Chance Encounters of the Musical Kind: Electronica and Audiovisual Synchronization in Three Films Directed by Tom Tykwer

Abstract This essay examines the interaction of electronica music, editing, and shot composition in three films directed by Tom Tykwer, all of which share the dramatic theme of the chance encounter, and all of which were scored by Tykwer in collaboration with pop-music composers and producers Johnny Klimek and Reinhold Heil. By using electronica music, the films train the audience’s attention to subtle changes of musical texture, which, when synchronized to changes in the image, help to convey the theme of the chance encounter. I argue further that the compositional practices that characterize Tykwer’s approach to film music constitute a fundamental shift not only from the leitmotif-bound tradition of classical Hollywood scoring but also from that same tradition’s Romantic conception of the single, creative artist who works within a musical canon associated with high art.

Determining the Amount of Audio-Video Synchronization Errors Perceptible to the Average End-User

The media and acoustics perception lab (MAPL) designed a study to determine the minimum amount of audio-visual synchronization (a/v sync) errors that can be detected by end-users. Lip synchronization is the most noticeable a/v sync error, and was used as the testing stimuli to determine the perceptual threshold of audio leading errors. The results of the experiment determined that the average audio leading threshold for a/v sync detection was 185.19 ms, with a standard deviation of 42.32 ms. This threshold determination of lip sync error (with audio leading) will be widely used for validation and verification infrastructures across the industry. By implementing an objective pass/fail value into software, the system or network under test is held against criteria which were derived from a scientific subjective test.

Difference between diotic and dichotic presentations on audiovisual synchronization

Significant differences were found between diotic and dichotic presentations of auditory stimuli on audiovisual synchronization. Two types of moving sphere were used for visual stimuli; (a) a sphere moved straight from left to right at a constant velocity, and (b) it moved up and right and then turned down and right at the midpoint. Auditory stimuli included (1) a continuous pure tone (1000 Hz), (2) a sequence of two short tones of 30 ms (1000 Hz), (3) a sweep tone (500 Hz → 1500 Hz), and (4) a Frequency-Modulated tone (500 Hz → 1500 Hz → 500 Hz). The duration of these stimuli was 240 ms each. Visual stimuli were presented on a Liquid-Crystal Display. Auditory stimuli were presented by headphones. The time difference between auditory and visual stimuli was changed between −360 ms and +360 ms. The sound source linearly moved from left to right in case of dichotic presentation. Subjective tests, using the forced-choice of synchronized or not, clarified the significant differences between diotic and dichotic presentations when the auditory stimulus (1) or (4) was presented with the visual stimulus (b). No significance was obtained for other combinations of auditory and visual stimuli. [Work supported by NTT.]

Synchronization of musical sound and visual images: Issues of empirical and practical significance in multimedia development

In recent years, there has been a significant number of investigations into the interaction between auditory and visual systems in a multi-modal context, e.g., film, animation, and multimedia software. The author will report results of his own research, confirming a subject preference for alignment of audio-visual (AV) accent structures, i.e., combinations resulting in perceived AV synchronization. A revised version of a model of Film Music Perception [Lipscomb and Kendall (1994)] will be presented, explicating a dynamic relationship between stimulus complexity and significance of AV synchronization in the determination of subject preference ratings. Using these findings as a starting point, the author will illustrate the practical importance of musical sound and its temporal placement within a multimedia context. Multimedia software and technologically enhanced course materials created by the author will provide examples of the effect of combining musical sound and visual images. The presentation will illustrate that the combinatorial effect of such multimedia presentations far surpasses the effect of either auditory or visual stimuli in isolation, resulting in an improved aesthetic experience for the audience and an enhanced learning environment for the student.

Computational Requirements and Synchronization Issues for Virtual Acoustic Displays

This paper addresses two main issues concerning virtual acoustic displays. First, we discuss the computational requirements including sound generation (or synthesis), environmental effects modeling, and three-dimensional (3-D) sound localization. The computational analysis reveals that acoustic processing delays of at least 66 ms are expected with today's technology. This analysis motivates the second issue: how much computational time is available for executing the acoustic process, assuming the requirement for perceptually perfect audiovisual synchronization? A psychoacoustic experiment designed to quantify the tolerable audiovisual delay indicates that an acoustic impact event must occur within an average of 175 ms of the visual event in order for the events to be perceived as synchronous. The most highly trained observers detect desynchrony with an audiovisual delay as low as 100 ms. The results of the computational requirement analysis and the psychoacoustic synchronization experiment provide important information for designers and researchers of virtual acoustic displays.

Studies on the motility of smooth muscles of the teats in lactating cows.

Rhythmic motility of teats in lactating cows in vivo was studied by an improved plethysmographic technique. Oxytocin was infused intravenously and drugs were injected into the artery of the filled udder. The beta-adrenergic agent, isoprenaline, induced relaxation of the teats whereas the alpha-adrenergic agents phenylephrine and noradrenaline, elicited contraction. In one animal however, noradrenaline always exerted a biphasic effect. Pre-treatment with propranolol, a beta-adrenergic blocking agent, changed the biphasic effect into a purely activating response. Adrenaline, which exhibits both alpha and beta effects, induced relaxation, but after propranolol treatment relaxation was abolished and even converted into contraction of the teat. It seems clear that both alpha and beta adrenoceptors are present in longitudinal smooth muscles of the teats. Milk losses as intermittent spurts were observed in several cows. Spurts occurred synchronously with rhythmical teat motility in the filled udder during oxytocin infusion. The intensity of milk leakage was increased by isoprenaline and inhibited by noradrenaline, indicating the presence of alpha and beta receptors in teat sphincter muscles. Starting and stopping of milk spurts in correlation with rhythmical teat contractions were studied by audiovisual synchronization of video recording tape. The most common type of spurt began near the middle of the teat's relaxation phase and ended either just before or exactly at the beginning of the contraction phase. Interruption of the spurt is primarily due to closure of the streak canal.