Automated Event Detection and Classification in Soccer: The Potential of Using Multiple Modalities

Olav Andre Nergård Rongved,Evi Zouganeli,Markus Stige,Cise Midoglu,Pål Halvorsen,Steven Alexander Hicks,Vajira Lasantha Thambawita,Michael Alexander Riegler,Dag Johansen

doi:10.3390/make3040051

Abstract

Detecting events in videos is a complex task, and many different approaches, aimed at a large variety of use-cases, have been proposed in the literature. Most approaches, however, are unimodal and only consider the visual information in the videos. This paper presents and evaluates different approaches based on neural networks where we combine visual features with audio features to detect (spot) and classify events in soccer videos. We employ model fusion to combine different modalities such as video and audio, and test these combinations against different state-of-the-art models on the SoccerNet dataset. The results show that a multimodal approach is beneficial. We also analyze how the tolerance for delays in classification and spotting time, and the tolerance for prediction accuracy, influence the results. Our experiments show that using multiple modalities improves event detection performance for certain types of events.

Highlights

The generation of video summaries and highlights from sports games is of tremendous interest for broadcasters as a large percent of audiences prefer to view only the main events in a game
Using the SoccerNet dataset [1], from which a number of samples are presented in Figure 1, we evaluate two previous unimodal approaches focusing on video [2,4] and experiment with different parameters, showing the trade-offs between accuracy and latency
We focus on action detection in the context of sports, and, soccer videos

Summary

Introduction

The generation of video summaries and highlights from sports games is of tremendous interest for broadcasters as a large percent of audiences prefer to view only the main events in a game. The “holy grail” of event annotation has long been anticipated to be a fully automated event extraction system Such a system needs to be timely and accurate, and must capture events in near real-time. One of the key components of such a comprehensive pipeline is the detection and classification of significant events in real-time Existing approaches for this task need significant improvement in terms of detection and classification accuracy. Much work has been done on using visual information to detect events in soccer videos, and promising examples with a good detection performance [1,2,3,4] are available. In addition to the visual ResNet features provided by SoccerNet, we extract information from the video and audio streams in the dataset directly.

Objectives

Results

Discussion

Conclusion