MIRRE approach: nonlinear and multimodal exploration of MIR aggregated search results

Abstract

Nowadays, web users frequently explore multimedia contents to satisfy their information needs. The exploration approaches usually provide linear interaction mechanisms and do not exploit the multiple information modalities associated with results. They cannot treat multimedia documents as aggregated entities. The aggregation of results in multimedia documents and nonlinear navigation in them is usually not possible. The exploration of multimedia content segregated in multiple verticals is tedious. In this research, we propose an approach to address the core issues in multimedia contents exploration. We provide a nonlinear and multimodal exploration of multimedia document results. We generate result spaces by exploiting multimodal similarity and semantic relationships in results and enable their nonlinear and multimodal exploration via a search user interface (SUI) design. The result space connects retrieved multimedia documents and their aggregated media objects via multimodal similarity and semantic relationships, respectively. The SUI provides lookup, preview/view/access, browsing, and visualization of multimedia documents by introducing various types of interface components. The approach has been instantiated over a real dataset of aggregated multimedia documents and evaluated via usability tests. Our investigation reveals that 93.33% users had completed exploration tasks within time limits. The overall usability scores assessed via SUS and PUEU instruments are 72.1% and 61.6%, respectively. The user satisfaction tested via QUIS-instrument is 70.3%. The usage of elementary SUI components providing the nonlinear exploration is high. The ploy-representation of results improves the overall information gain. The evaluation reveals that approach is usable, gives a satisfactory exploration mechanism, and SUI components collectively provide exploration.