Elsevier

Pattern Recognition Letters

Volume 125, 1 July 2019, Pages 527-533
Pattern Recognition Letters

Combining graph edit distance and triplet networks for offline signature verification

https://doi.org/10.1016/j.patrec.2019.06.024Get rights and content

Highlights

  • Combination of a structural and a statistical signature verification system.

  • The structural approach is based on an approximation of the graph edit distance.

  • The statistical approach is based on a convolutional neural network with triplet loss.

  • Combined approach improves significantly compared to individual approaches.

  • Excellent results on 4 public datasets despite being trained on synthetic data only.

Abstract

Offline signature verification is a challenging pattern recognition task where a writer model is inferred using only a small number of genuine signatures. A combination of complementary writer models can make it more difficult for an attacker to deceive the verification system. In this work, we propose to combine a recent structural approach based on graph edit distance with a statistical approach based on deep triplet networks. The combination of the structural and statistical models achieve significant improvements in performance on four publicly available benchmark datasets, highlighting their complementary perspectives.

Introduction

Handwritten signatures remain a widely used and accepted mean of biometric authentication even in the modern world. Hence, there is an interest in verifying the genuineness of signatures. To this day, automatic signature verification remains an active field of research [10], [19] and the levels of accuracy achieved by state-of-the-art systems is similar to that of other biometric verification systems [25]. The pattern recognition community is distinguishing between two cases of signature verification: the offline case, where only static images of the signatures are available, and the online case, where dynamic information like the velocity and pressure is additionally available.

The majority of current state-of-the-art approaches to offline signature verification use statistical pattern recognition, i.e. fixed-size feature vectors are used to represent signatures. In the past, these vector representations have been generated using handcrafted feature extractors, which leverage either local information, e.g. histogram of oriented gradients, local binary patterns, or Gaussian grid features taken from signature contours [51], or global information, such as number of branches in the skeleton, number of holes, geometrical features like Fourier descriptors, position of barycenter, moments, projections, distributions, tortuosities, directions, curvatures and chain codes [25], [37]. In recent years, however, with the rise of deep learning, the state of the art shifts toward learning features directly from fixed-size signature images using neural networks [10], [19]. These networks rely on convolutional neural network architectures (CNN) of various kinds [18], [39], [54].

Structural pattern recognition using graphs for pattern representation offers another way of approaching signature verification. Graphs provide a powerful representation formalism that can be beneficial for signature verification. For example, graphs could use nodes to represent local information and edges to model the nodes’ relation in the global structure.

The problem of graph dissimilarity computation is often solved via error-tolerant graph matching algorithm [9], [14]. One approach is to find a mapping that minimizes a given cost function. However, the problem of optimizing this cost is known to be NP-complete [53]. This means that the run time may be intractable even for rather small graphs, which may be one of the main reasons why graphs have rarely been used for signature verification in the past.

In recent years, however, several approximate, or suboptimal, algorithms for graph matching have been proposed [7], [27], [40], [53]. These algorithms offer polynomial, rather than exponential, run times. Yet, they do not guarantee to find the global minimum of the matching cost, but only a local one.

Two alternative families of error-tolerant graph matching that differ in their basis from more traditional approaches, are graph embeddings and graph kernels. An important class of graph embedding are Spectral methods [49], [50]. However, there are many other graph embedding methods, for example, methods based on entropy computations [20]. Graph kernels provide an implicit graph embedding. An important group are Random walk kernels that measure the similarity of two graphs by the number of random walks in both graphs that have all or some labels in common [3].

Some early works using graphs for signature verification are representations based on stroke primitives [43], a modular graph matching approach [5], and basic concepts of graph theory [15]. More recently, a general signature verification framework based on the graph edit distance between labeled graphs has been introduced by Maergner et al. [32]. In that work, the computational complexity of graph-based pattern analysis is reduced by employing the bipartite approximation framework proposed by Riesen and Bunke [40]. This approach has been combined with a complementary structural approach called inkball models in [30].

In the present paper, we present an extension of the work published in [31]. The original work investigated whether structural and statistical signature models have complimentary strength and hence work well together in a multiple classifier system (see Fig. 1). This has been done by combining an approach based on graph edit distance with a convolutional neural network using the triplet loss function [23]. We aim at further investigating this combination in this journal extension. The focus hereby is the practical application of the combined structural and statistical approach on a challenging real-world problem and a more comprehensive evaluation. Furthermore, we investigate possible improvements in neural network architecture and training.

Compared to the original publication, we utilized a more powerful network architecture, named DenseNet-121 [24], to compare against the previously used architecture, ResNet-18 [22]. The reason behind this choice is the particular nature of the DenseNet architecture, which allows features from lower layers to be propagated directly to the higher layers of the network. This is known to work well in natural images and in this work, we aim to find out whether this generalizes to the signature verification domain. Furthermore, we investigate an additional pretraining step for the neural networks in which we train for classification before training for similarity. This pretraining step has been shown to increase the network performances especially when for each class, there is little amount of labeled data available [38]. Finally, we are using two more test sets and more evaluation metrics to compare our framework against more published results in our experimental evaluation. That is, our evaluation is now performed on four publicly available datasets.

This paper is structured as follows. The graph-based approach is reviewed in Section 2 and the neural-network-based approach is described in Section 3. Eventually, the signature verification system using both approaches is detailed in Section 4. Finally, we present and discuss our experimental results in Section 5 and deduce our conclusions in Section 6.

Section snippets

Structural graph-based approach

Our structural approach for signature verification has been proposed in [32] and is based on graph edit distance. That is, the dissimilarity of two signatures is measured by comparing two keypoint graphs that are created from the corresponding signature images. In order to compute the graph edit distance, a suboptimal algorithm [40] is actually employed. The individual steps, viz. the graph extraction and graph comparison, are briefly described in the following two subsections. A more detailed

Statistical neural network-based approach

In the last decade, convolutional neural networks (CNN) have become state of the art in a large variety of applications, especially in computer-vision tasks. In fact, already back in 2012 CNNs have been proven to be suited to work with images [29]. Over the years there have been many developments and improvements and nowadays, CNNs represent the backbone of most vision-based applications. The idea of our neural network-based approach is to train a deep CNN to embed images of signatures into a

Combined signature verification system

A signature verification system calculates a dissimilarity score between reference signatures of the claimed user and an unseen signature. If this dissimilarity score (see Eqs. (6) or (7)) is below a certain threshold the signature is accepted as genuine, otherwise, the signature is rejected as a forgery.

Experimental evaluation

In this section, the experimental evaluation of our signature verification system is introduced. We describe the used datasets, the employed evaluation metrics, the training process, and finally, the results on four publicly available datasets are compared against the state of the art.

Conclusions and outlook

The performance of a signature verification system on four benchmark datasets is significantly improved when combining structural and statistical models. Individually, the structural model based on graph edit distance performs overall better on skilled forgeries, while the statistical model based on deep triplet networks performs significantly better on random forgeries. These complementary strengths have been combined in our proposed multiple classifier system. Overall, the system generalized

Declaration of Competing Interest

None.

Acknowledgment

This work has been supported by the Swiss National Science Foundation project 200021_162852.

References (54)

  • F. Alonso-Fernandez et al.

    Automatic measures for predicting performance in off-line signature

    Proc. of International Conference on Image Processing

    (2007)
  • V. Balntas et al.

    Learning local feature descriptors with triplets and shallow convolutional neural networks

  • A. Bansal et al.

    Offline signature verification using critical region matching

    Int. J. Signal Process. Image Process. Pattern

    (2009)
  • R.K. Bharathi et al.

    Off-line signature verification based on chain code histogram and support vector machine

    International Conference on Advances in Computing, Communications and Informatics (ICACCI)

    (2013)
  • M. Boeres et al.

    A randomized heuristic for scene recognition by graph matching

  • S. Chen et al.

    A new off-line signature verification method based on graph

    Proc. of International Conference on Pattern Recognition (ICPR)

    (2006)
  • D. Conte et al.

    Thirty years of graph matching in pattern recognition

    Int. J. Pattern Recognit. Art. Intell.

    (2004)
  • M. Diaz et al.

    A perspective analysis of handwritten signature technology

    ACM Comput. Surv.

    (2019)
  • M.A. Ferrer, GPDS database website, 2016. Accessed on Jan 28,...
  • M.A. Ferrer et al.

    Static signature synthesis: a neuromotor inspired approach for biometrics

    IEEE Trans. Pattern Anal. Mach.Intell.

    (2015)
  • J. Fierrez-Aguilar et al.

    An off-line signature verification system based on fusion of local and global information

    Biometric Authentication

    (2004)
  • P. Foggia et al.

    Graph matching and learning in pattern recognition in the last 10 years

    Int. J. Pattern Recognit. Art. Intell.

    (2014)
  • T. Fotak et al.

    Handwritten signature identification using basic concepts of graph theory

    WSEAS Trans. Signal Process.

    (2011)
  • A. Gilperez et al.

    Off-line signature verification using contour features

    Proc. of International Conference on Frontiers in Handwriting Recognition (ICFHR)

    (2008)
  • L.G. Hafemann et al.

    Fixed-sized representation learning from offline handwritten signatures of different sizes

    Int. J. Doc. Anal.Recognit. (IJDAR)

    (2018)
  • L.G. Hafemann et al.

    Offline handwritten signature verification - literature review

    Proc. of International Conference on Image Processing Theory, Tools and Applications (IPTA)

    (2017)
  • L. Han et al.

    Generative graph prototypes from information theory

    IEEE Trans. Pattern Anal. Mach.Intell.

    (2015)
  • Cited by (50)

    • From text to signatures: Knowledge transfer for efficient deep feature learning in offline signature verification

      2022, Expert Systems with Applications
      Citation Excerpt :

      The learning-based approaches seem to be more efficient in OSV task since the features are learnt directly from the images (Hafemann et al., 2017b; Zois, Tsourounis, Theodorakopoulos, Kesidis, & Economou, 2019). The most prominent classes of algorithms from this group are the methods that rely on learning a dictionary from signature images, while the images are subsequently encoded using the learned dictionaries (Zois et al., 2019; Zois, Theodorakopoulos, Tsourounis, & Economou, 2017; Zois, Theodorakopoulos, & Economou, 2017a; Zois, Papagiannopoulou, Tsourounis, & Economou, 2018) and methods based on deep learning (Gumusbas & Yildirim, 2019; Hafemann et al., 2017b, 2018; Maergner et al., 2019; Masoudnia et al., 2019; Yılmaz & Öztürk, 2020). The first approach of harnessing deep representations for OSV is, to the best of authors’ knowledge, the utilization of Restricted Boltzmann Machine for learning an encoding/representation function (Ribeiro, Gonçalves, Santos, & Kovacec, 2011).

    View all citing articles on Scopus
    View full text