Align While Fusion: A Generalized Nonaligned Multiview Multilabel Classification Method.

Abstract

In the task of multiview multilabel (MVML) classification, each object is described by several heterogeneous view features and annotated with multiple relevant labels. Existing MVML methods usually assume that these heterogeneous features are strictly view-aligned, and they directly conduct cross-view information fusion to train a multilabel prediction model. However, in real-world scenarios, such strict view-aligned requirement can be hardly satisfied due to the recurrent spatiotemporal asynchronism when collecting MVML data, which would cause inaccurate multiview fusion results and degrade the classification performance. To address this issue, we propose a generalized nonaligned MVML (GNAM) classification method, which achieves multiview information fusion while aligning cross-view features and accordingly learns a desired multilabel classifier. Specifically, we first introduce a multiorder matching alignment strategy to achieve cross-view feature alignments, where both first-order feature correspondence and second-order structure correspondence are jointly integrated to guarantee the compactness of the view-alignment results. Afterward, a commonality-and individuality-based multiview fusion structure is formulated on the aligned-view features to excavate the consistencies and complementarities across different views, which leads all relevant multiview semantic labels, especially rare labels, to be characterized more comprehensively. Finally, we embed adaptive global label correlations to multilabel classification model to further enhance its semantic expression integrity and develop an alternative algorithm to optimize the whole model. Extensive experimental results have verified that GNAM is significantly superior to other state-of-the-art methods.