Skip to main content
Log in

Benchmarking lightweight face architectures on specific face recognition scenarios

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

This paper studies the impact of lightweight face models on real applications. Lightweight architectures proposed for face recognition are analyzed and evaluated on different scenarios. In particular, we evaluate the performance of five recent lightweight architectures on five face recognition scenarios: image and video based face recognition, cross-factor and heterogeneous face recognition, as well as active authentication on mobile devices. In addition, we show the lacks of using common lightweight models unchanged for specific face recognition tasks, by assessing the performance of the original lightweight versions of the lightweight face models considered in our study. We also show that the inference time on different devices and the computational requirements of the lightweight architectures allows their use on real-time applications or computationally limited platforms. In summary, this paper can serve as a baseline in order to select lightweight face architectures depending on the practical application at hand. Besides, it provides some insights about the remaining challenges and possible future research topics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 12:2037–2041

    Article  Google Scholar 

  • Balaban S (2015) Deep learning and face recognition: the state of the art. Biomet Surveill Technol Human Act Identif 9457: 94570B. International Society for Optics and Photonics

  • Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces versus fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  • Cai H, Zhu L, Han S (2018) Proxylessnas: direct neural architecture search on target task and hardware. CoRR arXiv:1812.00332

  • Cao Q, Shen L, Xie W, Parkhi OM, Zisserman A (2018) Vggface2: a dataset for recognising faces across pose and age. In: 2018 13th IEEE international conference on automatic face and gesture recognition (FG 2018), pp 67–74. IEEE

  • Chen D, Ren S, Wei Y, Cao X, Sun J (2014) Joint cascade face detection and alignment. In: European conference on computer vision, pp 109–122. Springer

  • Chen JC, Zheng J, Patel VM, Chellappa R (2016) Fisher vector encoded deep convolutional features for unconstrained face verification. In: 2016 IEEE International conference on image processing (ICIP), pp 2981–2985. IEEE

  • Chen S, Liu Y, Gao X, Han Z (2018) Mobilefacenets: efficient cnns for accurate real-time face verification on mobile devices. In: Zhou J, Wang Y, Sun Z, Jia Z, Feng J, Shan S, Ubul K, Guo Z (eds), Biometric recognition, pp 428–438

  • Chen T, Li M, Li Y, Lin M, Wang N, Wang M, Xiao T, Xu B, Zhang C, Zhang Z (2015) Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems. CoRR arXiv:1512.01274

  • Cheng J, Wang PS, Li G, Hu QH, Lu HQ (2018) Recent advances in efficient computation of deep convolutional neural networks. Front Inform Technol Electr Eng 19(1):64–77

    Article  Google Scholar 

  • Courbariaux M, Bengio Y (2016) Binarynet: training deep neural networks with weights and activations constrained to +1 or -1. CoRR arXiv:1602.02830

  • Courbariaux M, Bengio Y, David J (2015) Binaryconnect: training deep neural networks with binary weights during propagations. CoRR arXiv:1511.00363

  • Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4690–4699

  • Deng J, Guo J, Zhang D, Deng Y, Lu X, Shi S (2019) Lightweight face recognition challenge. In: Proceedings of the ieee international conference on computer vision workshops

  • Deng J, Guo J, Zhou Y, Yu J, Kotsia I, Zafeiriou S (2019) Retinaface: single-stage dense face localisation in the wild. CoRR arXiv:1905.00641

  • Deng J, Zhou Y, Zafeiriou S (2017) Marginal loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 60–68

  • Denil M, Shakibi B, Dinh L, Ranzato M, De Freitas N (2013) Predicting parameters in deep learning. In: Advances in neural information processing systems, pp. 2148–2156

  • Ding C, Tao D (2017) Trunk-branch ensemble convolutional neural networks for video-based face recognition. IEEE Trans Pattern Anal Mach Intell 40:1002–1014

  • Duan Y, Lu J, Zhou J (2019) Uniformface: learning deep equidistributed representation for face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3415–3424

  • Duong CN, Luu K, Quach KG, Le N (2019) Shrinkteanet: million-scale lightweight face recognition via shrinking teacher-student networks. CoRR arXiv:1905.10620

  • Duong CN, Quach KG, Le N, Nguyen N, Luu K (2018) Mobiface: a lightweight deep learning face recognition on mobile devices. arXiv preprint arXiv:1811.11080

  • Fathy ME, Patel VM, Chellappa R (2015) Face-based active authentication on mobile devices. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1687–1691. IEEE

  • Galea C, Farrugia RA (2017) Matching software-generated sketches to face photographs with a very deep cnn, morphed faces, and transfer learning. IEEE Trans Inform Forensics Secur 13(6):1421–1431

    Article  Google Scholar 

  • Galea C, Farrugia RA (2018) Matching software-generated sketches to face photographs with a very deep cnn, morphed faces, and transfer learning. IEEE Trans Inform Forensics Sec 13(6):1421–1431. https://doi.org/10.1109/TIFS.2017.2788002

    Article  Google Scholar 

  • Grgic M, Delac K, Grgic S (2011) Scface-surveillance cameras face database. Multimed Tools Appl 51(3):863–879

    Article  Google Scholar 

  • Guo G, Zhang N (2019) A survey on deep learning based face recognition. Comput Vis Image Underst 189:102805

  • Guo Y, Zhang L, Hu Y, He X, Gao J (2016) Ms-celeb-1m: a dataset and benchmark for large-scale face recognition. CoRR arXiv:1607.08221

  • Haghighat M, Abdel-Mottaleb M (2017) Low resolution face recognition in surveillance systems using discriminant correlation analysis. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017), pp 912–917. IEEE

  • Han S, Mao H, Dally WJ (2015) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149

  • Harandi MT, Sanderson C, Shirazi S, Lovell BC (2011) Graph embedding discriminant analysis on grassmannian manifolds for improved image set matching. In: CVPR, p 2705–2712

  • He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: International conference on computer vision (ICCV), pp 1389–1397

  • Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

  • Howard A, Sandler M, Chu G, Chen LC, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, et al (2019) Searching for mobilenetv3. In: Proceedings of the IEEE international conference on computer vision, pp 1314–1324

  • Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  • Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  • Hu Q, Wang P, Cheng J (2018) From hashing to cnns: training binary weight networks via hashing. In: Thirty-Second AAAI conference on artificial intelligence

  • Huang G, Liu S, Van der Maaten L, Weinberger KQ (2018) Condensenet: An efficient densenet using learned group convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2752–2761

  • Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  • Huang GB, Ramesh M, Berg T, Learned-miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments

  • Huang Y, Shen P, Tai Y, Li S, Liu X, Li J, Huang F, Ji R (2020) Improving face recognition from hard samples via distribution distillation loss. arXiv preprint arXiv:2002.03662

  • Huang Z, Shan S, Wang R, Zhang H, Lao S, Kuerban A, Chen X (2015) A benchmark and comparative study of video-based face recognition on cox face database. IEEE Trans Image Process 24(12):5967–5981

    Article  MathSciNet  Google Scholar 

  • Huang Z, Wang R, Shan S, Chen X (2014) Learning euclidean-to-riemannian metric for point-to-set classification. In: CVPR, p 1677–1684

  • Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \(< 0.5\ \text{mb}\) model size. arXiv preprint arXiv:1602.07360

  • IBUG (2019) Lightweight face recognition challenge and workshop (ICCV 2019). https://ibug.doc.ic.ac.uk/. Accessed 2019 May 09

  • Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167

  • Jain AK, Li SZ (2011) Handbook of face recognition, vol 1, Springer

  • Kemelmacher-Shlizerman I, Seitz SM, Miller D, Brossard E (2016) The megaface benchmark: 1 million faces for recognition at scale. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4873–4882

  • Lebedev V, Ganin Y, Rakhuba M, Oseledets I, Lempitsky V (2014) Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553

  • Li F, Zhang B, Liu B (2016) Ternary weight networks. arXiv preprint arXiv:1605.04711

  • Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2016) Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710

  • Li X, Wang F, Hu Q, Leng C (2019) Airface:lightweight and efficient model for face recognition. In: The IEEE international conference on computer vision (ICCV) workshops

  • Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:1312.4400

  • Liu H, Simonyan K, Yang Y (2018) DARTS: differentiable architecture search. CoRR arXiv:1806.09055

  • Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) Sphereface: Deep hypersphere embedding for face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 212–220

  • Liu Y, Peng B, Shi P, Yan H, Zhou Y, Han B, Zheng Y, Lin C, Jiang J, Fan Y, et al (2018) iqiyi-vid: a large dataset for multi-modal person identification. arXiv preprint arXiv:1811.07548

  • Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE international conference on computer vision, pp 2736–2744

  • Llanes NM, Castillo-Rosado K, Méndez-Vázquez H, Khellat-Kihel S, Tistarelli M (2019) Face recognition on mobile devices based on frames selection. In: Progress in pattern recognition, image analysis, computer vision, and applications - 24th Iberoamerican congress, CIARP 2019, Havana, Cuba, October 28–31, 2019, Proceedings, pp 316–325

  • Lu Z, Jiang X, Kot A (2018) Deep coupled resnet for low-resolution face recognition. IEEE Signal Process Lett 25(4):526–530

    Article  Google Scholar 

  • Ma N, Zhang X, Zheng HT, Sun J (2018) Shufflenet v2: practical guidelines for efficient cnn architecture design. arXiv preprint arXiv:1807.11164

  • Mandal B, Lim RY, Dai P, Sayed MR, Li L, Lim JH (2016) Trends in machine and human face recognition. In: Advances in face detection and facial image analysis, pp 145–187. Springer

  • Martindez-Diaz Y, Luevano LS, Mendez-Vazquez H, Nicolas-Diaz M, Chang L, Gonzalez-Mendoza M (2019) Shufflefacenet: a lightweight face architecture for efficient and highly-accurate face recognition. In: The IEEE international conference on computer vision (ICCV) workshops

  • Martínez-Díaz Y, Chang L, Hernández N, Méndez-Vázquez H, Sucar LE (2016) Efficient video face recognition by using fisher vector encoding of binary features. In: ICPR, pp 1436–1441

  • Martínez-Díaz Y, Hernandez N, Biscay RJ, Chang L, Mendez-Vazquez H, Sucar LE (2018) On fisher vector encoding of binary features for video face recognition. J Vis Commun Image Represent 51:155–161

    Article  Google Scholar 

  • Martínez-Díaz Y, Méndez-Vázquez H, López-Avila L, Chang L, Enrique Sucar L, Tistarelli M (2018) Toward more realistic face recognition evaluation protocols for the youtube faces database. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 413–421

  • Maze B, Adams J, Duncan JA, Kalka N, Miller T, Otto C, Jain AK, Niggel WT, Anderson J, Cheney J, et al (2018) Iarpa janus benchmark-c: Face dataset and protocol. In: 2018 International conference on biometrics (ICB), pp 158–165. IEEE

  • Moschoglou S, Papaioannou A, Sagonas C, Deng J, Kotsia I, Zafeiriou S (2017) Agedb: the first manually collected, in-the-wild age database. pp 1997–2005. https://doi.org/10.1109/CVPRW.2017.250

  • Méndez-Vázquez H, Becerra-Riera F, Morales-González A, López-Avila L, Tistarelli M (2019) Local deep features for composite face sketch recognition. In: 2019 7th international workshop on biometrics and forensics (IWBF), pp 1–6. https://doi.org/10.1109/IWBF.2019.8739212

  • Ng HW, Winkler S (2014) A data-driven approach to cleaning large face datasets. In: 2014 IEEE international conference on image processing (ICIP), pp 343–347. IEEE

  • Park W, Kim D, Lu Y, Cho M (2019) Relational knowledge distillation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3967–3976

  • Parkhi OM, Simonyan K, Vedaldi A, Zisserman A (2014) A compact and discriminative face track descriptor. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1693–1700

  • Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. Br Mach Vis Conf 1:1–12

    Google Scholar 

  • Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: imagenet classification using binary convolutional neural networks. In: European conference on computer vision, pp 525–542. Springer

  • Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550

  • Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  • Schroff F, Kalenichenko D, Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823

  • Sengupta S, Chen J, Castillo C, Patel VM, Chellappa R, Jacobs DW (2016) Frontal to profile face verification in the wild. In: 2016 IEEE winter conference on applications of computer vision (WACV), pp 1–9. https://doi.org/10.1109/WACV.2016.7477558

  • Sepas-Moghaddam A, Pereira F, Correia PL (2019) Face recognition: a novel multi-level taxonomy based survey. arXiv preprint arXiv:1901.00713

  • Sun Y, Liang D, Wang X, Tang X (2015) Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873

  • Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  • Trillionpairs (2019) http://trillionpairs.deepglint.com/overview. Accessed: 2020-07-23

  • Taigman Y, Yang M, Ranzato M, Wolf L (2014) Closing the gap to human-level performance in face verification. Deepface. IEEE Comput Vis Pattern Recognit (CVPR) 5:6

    Google Scholar 

  • Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2820–2828

  • Tan M, Le QV (2019) Efficientnet: rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946

  • Thomee B, Shamma DA, Friedland G, Elizalde B, Ni K, Poland D, Borth D, Li LJ (2015) Yfcc100m: the new data in multimedia research. arXiv preprint arXiv:1503.01817

  • Tran LQ, Yin X, Liu X (2018) Representation learning by rotating your faces. IEEE Trans Pattern Anal Mach Intell 41:3007–3021

  • Tung F, Mori G (2019) Similarity-preserving knowledge distillation. In: Proceedings of the IEEE international conference on computer vision, pp 1365–1374

  • Turk M, Pentland A (1991) Eigenfaces for recognition. J Cognit Neurosci 3(1):71–86

    Article  Google Scholar 

  • Wang F, Cheng J, Liu W, Liu H (2018) Additive margin softmax for face verification. IEEE Signal Process Lett 25(7):926–930

    Article  Google Scholar 

  • Wang M, Deng W (2018) Deep face recognition: a survey. arXiv preprint arXiv:1804.06655

  • Wang P, Cheng J (2016) Accelerating convolutional neural networks for mobile applications. In: Proceedings of the 24th ACM international conference on multimedia, pp 541–545

  • Wang Q, Guo G (2019) Benchmarking deep learning techniques for face recognition. J Vis Commun Image Represent 65:102,663

    Article  Google Scholar 

  • Wang R, Guo H, Davis LS, Dai Q (2012) Covariance discriminative learning: a natural and efficient approach to image set classification. In: CVPR, p 2496–2503

  • Wang X, Wang S, Zhang S, Fu T, Shi H, Mei T (2018) Support vector guided softmax loss for face recognition. arXiv preprint arXiv:1812.11317

  • Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: European conference on computer vision, pp 499–515. Springer

  • Whitelam C, Taborsky E, Blanton A, Maze B, Adams J, Miller T, Kalka N, Jain AK, Duncan JA, Allen K, et al (2017) Iarpa janus benchmark-b face dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 90–98

  • Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: CVPR, pp 529–534

  • Wu B, Wan A, Yue X, Jin P, Zhao S, Golmant N, Gholaminejad A, Gonzalez J, Keutzer K (2017) Shift: A zero flop, zero parameter alternative to spatial convolutions. arXiv preprint arXiv:1711.08141

  • Wu X, He R, Sun Z, Tan T (2018) A light cnn for deep face representation with noisy labels. IEEE Trans Inform Forensics Secur 13(11):2884–2896

    Article  Google Scholar 

  • Xie S, Zheng H, Liu C, Lin L (2018) SNAS: stochastic neural architecture search. CoRR arXiv:1812.09926

  • Xie W, Shen L, Zisserman A (2018) Comparator networks. In: Proceedings of the European conference on computer vision (ECCV), pp 782–797

  • Xie W, Zisserman A (2018) Multicolumn networks for face recognition. arXiv preprint arXiv:1807.09192

  • Yan M, Zhao M, Xu Z, Zhang Q, Wang G, Su Z (2019) Vargfacenet: An efficient variable group convolutional neural network for lightweight face recognition. In: The IEEE international conference on computer vision (ICCV) workshops

  • Yang F, Yang W, Gao R, Liao Q (2017) Discriminative multidimensional scaling for low-resolution face recognition. IEEE Signal Process Lett 25(3):388–392

    Article  Google Scholar 

  • Yang J, Ren P, Chen D, Wen F, Li H, Hua G (2017) Neural aggregation network for video face recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), In press

  • Yin X, Tai Y, Huang Y, Liu X (2019) Fan: feature adaptation network for surveillance face recognition and normalization. arXiv preprint arXiv:1911.11680

  • Zagoruyko S, Komodakis N (2016) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928

  • Zeng D, Chen H, Zhao Q (2016) Towards resolution invariant face recognition in uncontrolled scenarios. In: 2016 International conference on biometrics (ICB), pp 1–8. IEEE

  • Zhang J (2019) Seesaw-net: convolution neural network with uneven group convolution. arXiv preprint arXiv:1905.03672

  • Zhang J (2019) Seesawfacenets: sparse and robust face verification model for mobile platform. arXiv preprint arXiv:1908.09124

  • Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  • Zhang Q, Li J, Yao M, Song L, Zhou H, Li Z, Meng W, Zhang X, Wang G (2019) Vargnet: variable group convolutional neural network for efficient embedded computing. arXiv preprint arXiv:1907.05653

  • Zhang X, Zhou X, Lin M, Sun J (2017) Shufflenet: an extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083

  • Zhang X, Zou J, He K, Sun J (2015) Accelerating very deep convolutional networks for classification and detection. IEEE Trans Pattern Anal Mach Intell 38(10):1943–1955

    Article  Google Scholar 

  • Zheng T, Deng W (2018) Cross-pose lfw: a database for studying crosspose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Technical report, pp 18–01

  • Zheng T, Deng W, Hu J (2017) Cross-age lfw: a database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197

  • Zhi-Peng F, Yan-Ning Z, Hai-Yan H (2014) Survey of deep learning in face recognition. In: 2014 International conference on orange technologies, pp 5–8. IEEE

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Heydi Méndez-Vázquez.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Martínez-Díaz, Y., Nicolás-Díaz, M., Méndez-Vázquez, H. et al. Benchmarking lightweight face architectures on specific face recognition scenarios. Artif Intell Rev 54, 6201–6244 (2021). https://doi.org/10.1007/s10462-021-09974-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-021-09974-2

Keywords

Navigation