Adaptive music resizing with stretching, cropping and insertion

Abstract

Content-aware music adaption, i.e. music resizing, in temporal constraints starts drawing attention from multimedia communities, because there are plenty of real-world scenarios, e.g. animation production and radio advertisement production. The goal of music resizing is to change the length of a music track to a user preferred length using a series of basic operations, e.g. compression, prolonging, cropping and insertion. The only existing music resizing approach so far, called LyDAR, is based on the lyrics analysis and just utilizes the compression operation to resize a music piece. As a result, LyDAR suffers from some limitations, e.g., it can neither prolong a music track nor compress music pieces with very small stretch rates. In this paper, we propose a content-aware music resizing framework, named MUSIZ. In general, MUSIZ outperforms LyDAR in three aspects: (a) Except for the compression operation, MUSIZ takes advantages of prolonging, cropping and insertion operations to handle the resizing requests of both compression and prolonging. (b) Observing the diversity of quality degradation for different segments, we propose the concept of stretch-resistance to measure the degree of quality degradation after a segment is stretched. The stretch-resistance is modeled based on both acoustical and lyrics features. (c) Cropping and insertion operations are utilized before stretching. We develop the contiguity-preservative cropping and insertion algorithms to remove and insert music segments while smoothing the abrupt change at the joint between the manipulated segments. Comprehensive user studies show that the music tracks resized by MUSIZ achieve better quality than those produced by existing approaches.