Synchronous spatio-temporal signature verification via Fusion Triplet Supervised Network

Abstract

Handwritten signature has been regarded as one of the most accepted and practical means of person verification since ancient times. Unlike existing methods to trace the signer only considering a single spatial or temporal information, in this paper, we propose to combine two forms of signature data (stroke images and sensor signals) to realize person verification using the novel Fusion Triplet Supervised Network (fuseTSN). First, the Electronic Handwritten Signature Acquisition System 1.0 (EHS-AS 1.0) connected to the Wacom tablet is developed to construct a synchronous static–dynamic Chinese signature dataset (SynCS) and to transform existing dynamic signature datasets into static–dynamic datasets. Then, the hybrid feature extractor (ResCNN-BiLSTM with attention) as the backbone is embedded into the fuseTSN model, mapping the images and time-series signal into a common hyperspace. Furthermore, the similarity of the input [Anchor, Positive, Test] is measured to produce the final output. Finally, the proposed fuseTSN is tested on three public datasets and also evaluated on our SynCS dataset comprised of 24000 unique triplet label samples. The results show that the accuracy of the fuseTSN model is at least 5.01% higher than that of a unimodal system with only stroke image or sensor data on the SynCS dataset and outperforms the current state-of-the-art multimodal methods for writer-independence (WI) verification on public datasets. In addition, we present the fuseTSN’s reliable judgment basis for spatial strokes in the heat-map and further find that the combination of time attributes (OA and OE) can significantly improve the model performance.