Skip to main content
SpringerLink
Log in

Lightweight network-based multi-modal feature fusion for face anti-spoofing

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Effectively identifying attacked faces is an urgent problem to be solved in the scenario of face recognition. As deep learning is applied to face anti-spoofing (FAS), many multi-modal approaches have been proven to be more efficient than single-modal, but at the same time, multi-modal approaches require a huge number of parameters and therefore result in high computation. To tackle this problem, a lightweight network-based multi-modal FAS model was proposed, which took patch-level images from multi-modal images (YCbCr, Depth, and IR) as the input to different branches, and designed a lightweight feature extraction module to solve the redundancy of feature maps extracted by the filters. Finally, an attention-based feature fusion module was constructed to fuse and classify the features extracted by each branch network. A great number of comparative experimental results demonstrated that this method greatly reduced parameters at a high accuracy. For example, the accuracy on single-modal datasets (Replay-Attack and CASIA-FASD) is 100%, and that on multi-modal dataset (CASIA-SURF) is 98.1269% (TPR@FPR = 10e−4) and 0.2232%. In addition, the backbone network has only 0.25 M parameters and 0.37 G FLOPs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Pereira, T.D., Anjos, A., Martino, J.M., Marcel, S.: LBP–TOP based countermeasure against face spoofing attacks. ACCV Worksh. (2012). https://doi.org/10.1007/978-3-642-37410-4_11

    Article  Google Scholar 

  2. Boulkenafet, Z., Komulainen, J., Hadid, A.: Face antispoofing using speeded-up robust features and fisher vector encoding. IEEE Signal Process. Lett. 24, 141–145 (2017). https://doi.org/10.1109/LSP.2016.2630740

    Article  Google Scholar 

  3. Yang, J., Lei, Z., Liao, S., Li, S.: Face liveness detection with component dependent descriptor. In: 2013 International Conference on Biometrics (ICB) (2013), pp. 1–6. https://doi.org/10.1109/ICB.2013.6612955

  4. Boulkenafet, Z., Komulainen, J., Hadid, A.: Face spoofing detection using colour texture analysis. IEEE Trans. Inf. Forensics Secur. 11, 1818–1830 (2016). https://doi.org/10.1109/TIFS.2016.2555286

    Article  Google Scholar 

  5. Li, X., Komulainen, J., Zhao, G., Yuen, P., Pietikäinen, M.: Generalized face anti-spoofing by detecting pulse from face videos. In: 2016 23rd International Conference on Pattern Recognition (ICPR) (2016), pp. 4244–4249. https://doi.org/10.1109/ICPR.2016.7900300

  6. Lagorio, A., Tistarelli, M., Cadoni, M., Fookes, C., Sridharan, S.: Liveness detection based on 3D face shape analysis. In: 2013 International Workshop on Biometrics and Forensics (IWBF) (2013), pp. 1–4. https://doi.org/10.1109/IWBF.2013.6547310

  7. Wang, Y., Nian, F., Li, T., Meng, Z., Wang, K.: Robust face anti-spoofing with depth information. J. Vis. Commun. Image Represent. 49, 332–337 (2017). https://doi.org/10.1016/j.jvcir.2017.09.002

  8. Liu, S., Lan, X., Yuen, P.: Remote photoplethysmography correspondence feature for 3D mask face presentation attack detection. ECCV (2018). https://doi.org/10.1007/978-3-030-01270-0_34

  9. Liu, Y., Stehouwer, J., Jourabloo, A., Liu, X.: Deep tree learning for zero-shot face anti-spoofing. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), pp. 4675–4684. https://doi.org/10.1109/CVPR.2019.00481

  10. Zhang, S., Wang, X., Liu, A., Zhao, C., Wan, J., Escalera, S., Shi, H., Wang, Z., Li, S.: A dataset and benchmark for large-scale multi-modal face anti-spoofing. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), pp. 919–928. https://doi.org/10.1109/CVPR.2019.00101

  11. Xin, J. et al.: Facial attribute capsules for noise face super resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07 (2020)

  12. Xin, J., et al.: Video face super-resolution with motion-adaptive feedback cell. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07 (2020)

  13. Parkin, A., Grinchuk, O.: Recognizing multi-modal face spoofing with face recognition networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2019), pp. 1617–1623. https://doi.org/10.1109/CVPRW.2019.00204

  14. Shen, T., Huang, Y., Tong, Z.: FaceBagNet: Bag-of-local-features model for multi-modal face anti-spoofing. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2019), pp. 1611–1616. https://doi.org/10.1109/CVPRW.2019.00203

  15. Yu, Z., Zhao, C., Wang, Z., Qin, Y., Su, Z., Li, X., Zhou, F., Zhao, G.: Searching central difference convolutional networks for face anti-spoofing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), pp. 5294–5304. https://doi.org/10.1109/CVPR42600.2020.00534

  16. Yu, Z., Qin, Y., Li, X., Wang, Z., Zhao, X., Lei, Z., Zhao, G: Multi-modal face anti-spoofing based on central difference networks. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020), pp. 2766–2774. https://doi.org/10.1109/CVPRW50498.2020.00333

  17. Yu, Z., Wan, J., Qin, Y., Li, X., Li, S., Zhao, G.: NAS-FAS: Static-dynamic central difference network search for face anti-spoofing. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2020.3036338

  18. Peng, C., Wang, N., Li, J., Gao, X.: DLFace: Deep local descriptor for cross-modality face recognition. Pattern Recognit. 90, 161–171 (2019). https://doi.org/10.1016/j.patcog.2019.01.041

    Article  Google Scholar 

  19. Peng, C., Wang, N., Li, J., Gao, X.: Re-ranking high-dimensional deep local representation for NIR-VIS face recognition. IEEE Trans. Image Process. 28, 4553–4565 (2019). https://doi.org/10.1109/TIP.2019.2912360

    Article  MathSciNet  MATH  Google Scholar 

  20. Xin, J., Wang, N., Gao, X., Li, J.: Residual attribute attention network for face image super-resolution. AAAI (2019). https://doi.org/10.1609/aaai.v33i01.33019054

    Article  Google Scholar 

  21. Jiang, X., Wang, N., Xin, J., Yang, X., Yu, Y., Gao, X.: Image super-resolution via multi-view information fusion networks. Neurocomputing 402, 29–37 (2020). https://doi.org/10.1016/j.neucom.2020.03.073

    Article  Google Scholar 

  22. George, A., Marcel, S.: Cross modal focal loss for RGBD face anti-spoofing. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7878–7887 (2021). https://doi.org/10.1109/CVPR46437.2021.00779

  23. Liu, A., Tan, Z., Li, X., Wan, J., Escalera, S., Guo, G., Li, S.: Static and dynamic fusion for multi-modal cross-ethnicity face anti-spoofing. ArXiv: abs/1912.02340 (2019)

  24. Liu, A., Tan, Z., Li, X., Wan, J., Escalera, S., Guo, G., Li, S.Z.: CASIA-SURF CeFA: A benchmark for multi-modal cross-ethnicity face anti-spoofing. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2021), pp. 1178–1186. https://doi.org/10.1109/WACV48630.2021.00122

  25. Wang, Z., Yu, Z., Zhao, C., Zhu, X., Qin, Y., Zhou, Q., Zhou, F., Lei, Z.: Deep spatial gradient and temporal depth learning for face anti-spoofing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), pp. 5041–5050. https://doi.org/10.1109/CVPR42600.2020.00509

  26. Zhang, P., Zou, F., Wu, Z., Dai, N., Mark, S., Fu, M., Zhao, J., Li, K.: FeatherNets: convolutional neural networks as light as feather for face anti-spoofing. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2019), pp. 1574–1583. https://doi.org/10.1109/CVPRW.2019.00199

  27. Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.-C.: MobileNetV2: inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), pp. 4510–4520. https://doi.org/10.1109/CVPR.2018.00474

  28. Hu, J., Shen, L., Albanie, S., Sun, G., Wu, E.: Squeeze-and-excitation networks. IEEE Trans. Pattern Anal. Mach. Intell. (2020). https://doi.org/10.1109/TPAMI.2019.2913372

    Article  Google Scholar 

  29. Ma, N., Zhang, X., Zheng, H.-T., Sun, J.: ShuffleNet V2: Practical guidelines for efficient CNN architecture design. ECCV (2018). https://doi.org/10.1007/978-3-030-01264-9_8

  30. Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., & Xu, C.: GhostNet: more features from cheap operations. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), pp. 1577–1586. https://doi.org/10.1109/CVPR42600.2020.00165

  31. Wu, B., Wan, A., Yue, X., Jin, P.H., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J.E., Keutzer, K.: Shift: a zero FLOP, zero parameter alternative to spatial convolutions. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), pp. 9127–9135. https://doi.org/10.1109/CVPR.2018.00951

  32. Deng, J., Guo, J., Zafeiriou, S.: ArcFace: additive angular margin loss for deep face recognition. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), pp. 4685–4694. https://doi.org/10.1109/CVPR.2019.00482

  33. Chingovska, I., Anjos, A., Marcel, S.: On the effectiveness of local binary patterns in face anti-spoofing. In: 2012 BIOSIG: Proceedings of the International Conference of Biometrics Special Interest Group (BIOSIG), pp. 1–7 (2012)

  34. Zhang, Z., Yan, J., Liu, S., Lei, X., Yi, D., Li, S.: A face antispoofing database with diverse attacks. In: 2012 5th IAPR International Conference on Biometrics (ICB) (2012), pp. 26–31. https://doi.org/10.1109/ICB.2012.6199754

  35. Liu, Y., Jourabloo, A., Liu, X.: Learning deep models for face anti-spoofing: binary or auxiliary supervision. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), pp. 389–398. https://doi.org/10.1109/CVPR.2018.00048

  36. Kuang, H., Ji, R., Liu, H., Zhang, S., Sun, X., Huang, F., & Zhang, B. (2019). Multi-modal multi-layer fusion network with average binary center loss for face anti-spoofing. In: Proceedings of the 27th ACM International Conference on Multimedia. https://doi.org/10.1145/3343031.3351001

  37. Arora, S., Bhatia, M.P., Mittal, V.: A robust framework for spoofing detection in faces using deep learning. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02123-4

    Article  Google Scholar 

  38. Xu, Z., Li, S., Deng, W.: Learning temporal features using LSTM-CNN architecture for face anti-spoofing. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR) (2015), pp. 141–145. https://doi.org/10.1109/ACPR.2015.7486482

  39. Atoum, Y., Liu, Y., Jourabloo, A., Liu, X.: Face anti-spoofing using patch and depth-based CNNs. In: 2017 IEEE International Joint Conference on Biometrics (IJCB) (2017), pp. 319–328. https://doi.org/10.1109/BTAS.2017.8272713

  40. Li, L., Feng, X., Boulkenafet, Z., Xia, Z., Li, M., Hadid, A.: An original face anti-spoofing approach using partial convolutional neural network. In: 2016 sixth international conference on image processing theory, tools and applications (IPTA) (2016), pp. 1–6. https://doi.org/10.1109/IPTA.2016.7821013

  41. Sun, Y., Xiong, H., Yiu, S.: Understanding deep face anti-spoofing: from the perspective of data. Vis. Comput. 1–14 (2020). https://doi.org/10.1007/s00371-020-01849-x

  42. Komulainen, J., Hadid, A., Pietikäinen, M., Anjos, A., Marcel, S.: Complementary countermeasures for detecting scenic face spoofing attacks. In: 2013 International Conference on Biometrics (ICB) (2013), pp. 1–7. https://doi.org/10.1109/ICB.2013.6612968

  43. Khammari, M.: Robust face anti-spoofing using CNN with LBP and WLD. IET Image Process. 13, 1880–1884 (2019). https://doi.org/10.1049/iet-ipr.2018.5560

    Article  Google Scholar 

  44. Yang, X., Luo, W., Bao, L., Gao, Y., Gong, D., Zheng, S., Li, Z., & Liu, W.: Face anti-spoofing: model matters, so does data. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3502–3511 (2019). https://doi.org/10.1109/CVPR.2019.00362

  45. Li, X., Wan, J., Jin, Y., Liu, A., Guo, G., Li, S.: 3DPC-Net: 3D point cloud network for face anti-spoofing. In: 2020 IEEE International Joint Conference on Biometrics (IJCB) (2020), pp. 1–8. https://doi.org/10.1109/IJCB48548.2020.9304873

  46. Jourabloo, A., Liu, Y., Liu, X.: Face de-spoofing: anti-spoofing via noise modeling. ECCV (2018). https://doi.org/10.1007/978-3-030-01261-8_18

    Article  Google Scholar 

  47. Yang, J., Lei, Z., Li, S.: Learn convolutional neural network for face anti-spoofing. ArXiv: abs/1408.5601 (2014). https://doi.org/10.1007/978-3-319-21963-9_34

Download references

Acknowledgments

This work was supported by the Science and Technology Research Project of Chongqing Education Commission (Grant No. KJQN201900833), and the Scientific Research and Innovation Foundation of Chongqing, China (Grant No. CYS21398).

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Chongqing Technology and Business University, Chongqing, 400067, China

    Dan He, Xiping He, Rui Yuan & Yue Li

  2. Chongqing Engineering Laboratory for Detection, Control and Integrated System, Chongqing Technology and Business University, Chongqing, 400067, China

    Xiping He

  3. School of Management Science and Engineering, Chongqing Technology and Business University, Chongqing, 400067, China

    Chao Shen

Authors
  1. Dan He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiping He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rui Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yue Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chao Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiping He.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, D., He, X., Yuan, R. et al. Lightweight network-based multi-modal feature fusion for face anti-spoofing. Vis Comput 39, 1423–1435 (2023). https://doi.org/10.1007/s00371-022-02420-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02420-6

Keywords

Search

Navigation