Combining graph edit distance and triplet networks for offline signature verification
Introduction
Handwritten signatures remain a widely used and accepted mean of biometric authentication even in the modern world. Hence, there is an interest in verifying the genuineness of signatures. To this day, automatic signature verification remains an active field of research [10], [19] and the levels of accuracy achieved by state-of-the-art systems is similar to that of other biometric verification systems [25]. The pattern recognition community is distinguishing between two cases of signature verification: the offline case, where only static images of the signatures are available, and the online case, where dynamic information like the velocity and pressure is additionally available.
The majority of current state-of-the-art approaches to offline signature verification use statistical pattern recognition, i.e. fixed-size feature vectors are used to represent signatures. In the past, these vector representations have been generated using handcrafted feature extractors, which leverage either local information, e.g. histogram of oriented gradients, local binary patterns, or Gaussian grid features taken from signature contours [51], or global information, such as number of branches in the skeleton, number of holes, geometrical features like Fourier descriptors, position of barycenter, moments, projections, distributions, tortuosities, directions, curvatures and chain codes [25], [37]. In recent years, however, with the rise of deep learning, the state of the art shifts toward learning features directly from fixed-size signature images using neural networks [10], [19]. These networks rely on convolutional neural network architectures (CNN) of various kinds [18], [39], [54].
Structural pattern recognition using graphs for pattern representation offers another way of approaching signature verification. Graphs provide a powerful representation formalism that can be beneficial for signature verification. For example, graphs could use nodes to represent local information and edges to model the nodes’ relation in the global structure.
The problem of graph dissimilarity computation is often solved via error-tolerant graph matching algorithm [9], [14]. One approach is to find a mapping that minimizes a given cost function. However, the problem of optimizing this cost is known to be NP-complete [53]. This means that the run time may be intractable even for rather small graphs, which may be one of the main reasons why graphs have rarely been used for signature verification in the past.
In recent years, however, several approximate, or suboptimal, algorithms for graph matching have been proposed [7], [27], [40], [53]. These algorithms offer polynomial, rather than exponential, run times. Yet, they do not guarantee to find the global minimum of the matching cost, but only a local one.
Two alternative families of error-tolerant graph matching that differ in their basis from more traditional approaches, are graph embeddings and graph kernels. An important class of graph embedding are Spectral methods [49], [50]. However, there are many other graph embedding methods, for example, methods based on entropy computations [20]. Graph kernels provide an implicit graph embedding. An important group are Random walk kernels that measure the similarity of two graphs by the number of random walks in both graphs that have all or some labels in common [3].
Some early works using graphs for signature verification are representations based on stroke primitives [43], a modular graph matching approach [5], and basic concepts of graph theory [15]. More recently, a general signature verification framework based on the graph edit distance between labeled graphs has been introduced by Maergner et al. [32]. In that work, the computational complexity of graph-based pattern analysis is reduced by employing the bipartite approximation framework proposed by Riesen and Bunke [40]. This approach has been combined with a complementary structural approach called inkball models in [30].
In the present paper, we present an extension of the work published in [31]. The original work investigated whether structural and statistical signature models have complimentary strength and hence work well together in a multiple classifier system (see Fig. 1). This has been done by combining an approach based on graph edit distance with a convolutional neural network using the triplet loss function [23]. We aim at further investigating this combination in this journal extension. The focus hereby is the practical application of the combined structural and statistical approach on a challenging real-world problem and a more comprehensive evaluation. Furthermore, we investigate possible improvements in neural network architecture and training.
Compared to the original publication, we utilized a more powerful network architecture, named DenseNet-121 [24], to compare against the previously used architecture, ResNet-18 [22]. The reason behind this choice is the particular nature of the DenseNet architecture, which allows features from lower layers to be propagated directly to the higher layers of the network. This is known to work well in natural images and in this work, we aim to find out whether this generalizes to the signature verification domain. Furthermore, we investigate an additional pretraining step for the neural networks in which we train for classification before training for similarity. This pretraining step has been shown to increase the network performances especially when for each class, there is little amount of labeled data available [38]. Finally, we are using two more test sets and more evaluation metrics to compare our framework against more published results in our experimental evaluation. That is, our evaluation is now performed on four publicly available datasets.
This paper is structured as follows. The graph-based approach is reviewed in Section 2 and the neural-network-based approach is described in Section 3. Eventually, the signature verification system using both approaches is detailed in Section 4. Finally, we present and discuss our experimental results in Section 5 and deduce our conclusions in Section 6.
Section snippets
Structural graph-based approach
Our structural approach for signature verification has been proposed in [32] and is based on graph edit distance. That is, the dissimilarity of two signatures is measured by comparing two keypoint graphs that are created from the corresponding signature images. In order to compute the graph edit distance, a suboptimal algorithm [40] is actually employed. The individual steps, viz. the graph extraction and graph comparison, are briefly described in the following two subsections. A more detailed
Statistical neural network-based approach
In the last decade, convolutional neural networks (CNN) have become state of the art in a large variety of applications, especially in computer-vision tasks. In fact, already back in 2012 CNNs have been proven to be suited to work with images [29]. Over the years there have been many developments and improvements and nowadays, CNNs represent the backbone of most vision-based applications. The idea of our neural network-based approach is to train a deep CNN to embed images of signatures into a
Combined signature verification system
A signature verification system calculates a dissimilarity score between reference signatures of the claimed user and an unseen signature. If this dissimilarity score (see Eqs. (6) or (7)) is below a certain threshold the signature is accepted as genuine, otherwise, the signature is rejected as a forgery.
Experimental evaluation
In this section, the experimental evaluation of our signature verification system is introduced. We describe the used datasets, the employed evaluation metrics, the training process, and finally, the results on four publicly available datasets are compared against the state of the art.
Conclusions and outlook
The performance of a signature verification system on four benchmark datasets is significantly improved when combining structural and statistical models. Individually, the structural model based on graph edit distance performs overall better on skilled forgeries, while the statistical model based on deep triplet networks performs significantly better on random forgeries. These complementary strengths have been combined in our proposed multiple classifier system. Overall, the system generalized
Declaration of Competing Interest
None.
Acknowledgment
This work has been supported by the Swiss National Science Foundation project 200021_162852.
References (54)
- et al.
Quantum kernels for unattributed graphs using discrete-time quantum walks
Pattern Recognit. Lett.
(2017) - et al.
Learning features for offline handwritten signature verification using deep convolutional neural networks
Pattern Recognit.
(2017) - et al.
Image-based handwritten signature verification using hybrid methods of discrete radon transform, principal component analysis and probabilistic neural network
Appl. Soft Comput.
(2016) - et al.
Automatic signature verification and writer identification - the state of the art
Pattern Recognit.
(1989) - et al.
Approximate graph edit distance computation by means of bipartite graph matching
Image Vis. Comput.
(2009) - et al.
Deep multitask metric learning for offline signature verification
Pattern Recognit. Lett.
(2016) - et al.
Off-line signature verification based on grey level information using texture features
Pattern Recognit.
(2011) - et al.
Graph characteristics from the heat kernel trace
Pattern Recognit.
(2009) - et al.
Geometric characterization and clustering of graphs using heat kernel embeddings
Image Vis. Comput.
(2010) - et al.
DeepDIVA: a highly-functional python framework for reproducible experiments
International Conference on Frontiers in Handwriting Recognition (ICFHR), Niagara Falls, USA
(2018)
Automatic measures for predicting performance in off-line signature
Proc. of International Conference on Image Processing
Learning local feature descriptors with triplets and shallow convolutional neural networks
Offline signature verification using critical region matching
Int. J. Signal Process. Image Process. Pattern
Off-line signature verification based on chain code histogram and support vector machine
International Conference on Advances in Computing, Communications and Informatics (ICACCI)
A randomized heuristic for scene recognition by graph matching
A new off-line signature verification method based on graph
Proc. of International Conference on Pattern Recognition (ICPR)
Thirty years of graph matching in pattern recognition
Int. J. Pattern Recognit. Art. Intell.
A perspective analysis of handwritten signature technology
ACM Comput. Surv.
Static signature synthesis: a neuromotor inspired approach for biometrics
IEEE Trans. Pattern Anal. Mach.Intell.
An off-line signature verification system based on fusion of local and global information
Biometric Authentication
Graph matching and learning in pattern recognition in the last 10 years
Int. J. Pattern Recognit. Art. Intell.
Handwritten signature identification using basic concepts of graph theory
WSEAS Trans. Signal Process.
Off-line signature verification using contour features
Proc. of International Conference on Frontiers in Handwriting Recognition (ICFHR)
Fixed-sized representation learning from offline handwritten signatures of different sizes
Int. J. Doc. Anal.Recognit. (IJDAR)
Offline handwritten signature verification - literature review
Proc. of International Conference on Image Processing Theory, Tools and Applications (IPTA)
Generative graph prototypes from information theory
IEEE Trans. Pattern Anal. Mach.Intell.
Cited by (50)
Subscripto multiplex: A Riemannian symmetric positive definite strategy for offline signature verification
2023, Pattern Recognition LettersA novel way to formalize stable graph cores by using matching-graphs
2022, Pattern RecognitionDynamic and static feature fusion for increased accuracy in signature verification
2022, Signal Processing: Image CommunicationFrom text to signatures: Knowledge transfer for efficient deep feature learning in offline signature verification
2022, Expert Systems with ApplicationsCitation Excerpt :The learning-based approaches seem to be more efficient in OSV task since the features are learnt directly from the images (Hafemann et al., 2017b; Zois, Tsourounis, Theodorakopoulos, Kesidis, & Economou, 2019). The most prominent classes of algorithms from this group are the methods that rely on learning a dictionary from signature images, while the images are subsequently encoded using the learned dictionaries (Zois et al., 2019; Zois, Theodorakopoulos, Tsourounis, & Economou, 2017; Zois, Theodorakopoulos, & Economou, 2017a; Zois, Papagiannopoulou, Tsourounis, & Economou, 2018) and methods based on deep learning (Gumusbas & Yildirim, 2019; Hafemann et al., 2017b, 2018; Maergner et al., 2019; Masoudnia et al., 2019; Yılmaz & Öztürk, 2020). The first approach of harnessing deep representations for OSV is, to the best of authors’ knowledge, the utilization of Restricted Boltzmann Machine for learning an encoding/representation function (Ribeiro, Gonçalves, Santos, & Kovacec, 2011).