Audio and Music Analysis on the Web using Essentia.js

Albin Correya,Pablo Alonso-Jiménez,Dmitry Bogdanov,Jorge Marcos-Fernández,Luis Joglar-Ongay,Xavier Serra

doi:10.5334/tismir.111

Albin Correya, Pablo Alonso-Jiménez + Show 4 more

Open Access

https://doi.org/10.5334/tismir.111

Copy DOI

Abstract

Open-source software libraries have a significant impact on the development of Audio Signal Processing and Music Information Retrieval (MIR) systems. Despite the abundance of such tools, there is a lack of an extensive and easy-to-use reference library for audio feature extraction on Web clients. In this article, we present Essentia.js, an open-source JavaScript (JS) library for audio and music analysis on both web clients and JS engines. Along with the Web Audio API, it can be used for both offline and real-time audio feature extraction on web browsers. Essentia.js is modular, lightweight, and easy-to-use, deploy, maintain, and integrate into the existing plethora of JS libraries and web technologies. It is powered by a WebAssembly back end cross-compiled from the Essentia C++ library, which facilitates a JS interface to a wide range of low-level and high-level audio features, including signal processing MIR algorithms as well as pre-trained TensorFlow.js machine learning models. It also provides a higher-level JS API and add-on MIR utility modules along with extensive documentation, usage examples, and tutorials. We benchmark the proposed library on two popular web browsers and the Node.js engine, and four devices, including mobile Android and iOS, comparing it to the native performance of Essentia and the Meyda JS library.

Highlights

The Web has become one of the most used computing platforms with billions of web pages served daily, and JavaScript (JS) is an essential part of building modern web applications
With the adoption of both HTML5 and the W3C Web Audio API specifications (Adenot and Choi, 2021), modern web browsers are capable of audio processing, synthesis, and analysis without any third-party dependencies on
It is based on the Essentia C++ library commonly used in Music Information Retrieval (MIR), which we ported to JS via Wasm and made compatible with the latest technologies in the Web Audio ecosystem

Summary

Introduction

The Web has become one of the most used computing platforms with billions of web pages served daily, and JavaScript (JS) is an essential part of building modern web applications. Other attempts at bringing audio processing and feature extraction onto the web client side have been made by using Emscripten to cross-compile tools written in other languages to JS via WebAssembly (e.g., Piper, Faust, and CsoundEmscripten) (Thompson et al, 2017; Letz et al, 2017; Bernardo et al, 2019; Lazzarini et al, 2014; 2015) Still, these tools are limited in the number of audio analysis features available out of the box, especially for MIR tasks. We present Essentia.js, an open-source JS library for audio and music analysis on the web, released under the AGPLv3 license It allows audio analysis and MIR applications to be built for web browsers and JS engines such as Node.js. Numerous web applications were built for aiding tasks such as crowdsourcing audio annotations (Fonseca et al, 2017; Cartwright et al, 2017), audio listening tests (Schoeffler et al, 2015; Jillings et al, 2015), and music

Last updated

Model MusiCNN VGG VGGish TempoCNN

Findings

Conclusions