Musical Hyperlapse: A Multimodal Approach to Accelerate First-Person Videos

Diognei de Matos, Washington Ramos, Luiz Romanhol, Erickson R. Nascimento

Abstract

With the advance of technology and social media usage, the recording of first-person videos is a widespread habit. These videos are usually very long and tiring to watch, bringing the need to speed-up them. Despite recent progress of fast-forward methods, in general, they do not consider inserting background music in the videos, which could make them more enjoyable. This paper presents a new methodology that creates accelerated videos and includes the background music keeping the same emotion induced by visual and acoustic modalities. Our methodology is based on the automatic recognition of emotions induced by music and video contents and an optimization algorithm that maximizes the visual quality of the output video and seeks to match the similarity of the music and the video’s emotions. Quantitative results show that our method achieves the best performance in matching emotion similarity while also maintaining the visual quality of the hyperlapse when compared with other literature methods.