Abstract
This paper studies the impact of lightweight face models on real applications. Lightweight architectures proposed for face recognition are analyzed and evaluated on different scenarios. In particular, we evaluate the performance of five recent lightweight architectures on five face recognition scenarios: image and video based face recognition, cross-factor and heterogeneous face recognition, as well as active authentication on mobile devices. In addition, we show the lacks of using common lightweight models unchanged for specific face recognition tasks, by assessing the performance of the original lightweight versions of the lightweight face models considered in our study. We also show that the inference time on different devices and the computational requirements of the lightweight architectures allows their use on real-time applications or computationally limited platforms. In summary, this paper can serve as a baseline in order to select lightweight face architectures depending on the practical application at hand. Besides, it provides some insights about the remaining challenges and possible future research topics.
Similar content being viewed by others
References
Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 12:2037–2041
Balaban S (2015) Deep learning and face recognition: the state of the art. Biomet Surveill Technol Human Act Identif 9457: 94570B. International Society for Optics and Photonics
Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces versus fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720
Cai H, Zhu L, Han S (2018) Proxylessnas: direct neural architecture search on target task and hardware. CoRR arXiv:1812.00332
Cao Q, Shen L, Xie W, Parkhi OM, Zisserman A (2018) Vggface2: a dataset for recognising faces across pose and age. In: 2018 13th IEEE international conference on automatic face and gesture recognition (FG 2018), pp 67–74. IEEE
Chen D, Ren S, Wei Y, Cao X, Sun J (2014) Joint cascade face detection and alignment. In: European conference on computer vision, pp 109–122. Springer
Chen JC, Zheng J, Patel VM, Chellappa R (2016) Fisher vector encoded deep convolutional features for unconstrained face verification. In: 2016 IEEE International conference on image processing (ICIP), pp 2981–2985. IEEE
Chen S, Liu Y, Gao X, Han Z (2018) Mobilefacenets: efficient cnns for accurate real-time face verification on mobile devices. In: Zhou J, Wang Y, Sun Z, Jia Z, Feng J, Shan S, Ubul K, Guo Z (eds), Biometric recognition, pp 428–438
Chen T, Li M, Li Y, Lin M, Wang N, Wang M, Xiao T, Xu B, Zhang C, Zhang Z (2015) Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems. CoRR arXiv:1512.01274
Cheng J, Wang PS, Li G, Hu QH, Lu HQ (2018) Recent advances in efficient computation of deep convolutional neural networks. Front Inform Technol Electr Eng 19(1):64–77
Courbariaux M, Bengio Y (2016) Binarynet: training deep neural networks with weights and activations constrained to +1 or -1. CoRR arXiv:1602.02830
Courbariaux M, Bengio Y, David J (2015) Binaryconnect: training deep neural networks with binary weights during propagations. CoRR arXiv:1511.00363
Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4690–4699
Deng J, Guo J, Zhang D, Deng Y, Lu X, Shi S (2019) Lightweight face recognition challenge. In: Proceedings of the ieee international conference on computer vision workshops
Deng J, Guo J, Zhou Y, Yu J, Kotsia I, Zafeiriou S (2019) Retinaface: single-stage dense face localisation in the wild. CoRR arXiv:1905.00641
Deng J, Zhou Y, Zafeiriou S (2017) Marginal loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 60–68
Denil M, Shakibi B, Dinh L, Ranzato M, De Freitas N (2013) Predicting parameters in deep learning. In: Advances in neural information processing systems, pp. 2148–2156
Ding C, Tao D (2017) Trunk-branch ensemble convolutional neural networks for video-based face recognition. IEEE Trans Pattern Anal Mach Intell 40:1002–1014
Duan Y, Lu J, Zhou J (2019) Uniformface: learning deep equidistributed representation for face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3415–3424
Duong CN, Luu K, Quach KG, Le N (2019) Shrinkteanet: million-scale lightweight face recognition via shrinking teacher-student networks. CoRR arXiv:1905.10620
Duong CN, Quach KG, Le N, Nguyen N, Luu K (2018) Mobiface: a lightweight deep learning face recognition on mobile devices. arXiv preprint arXiv:1811.11080
Fathy ME, Patel VM, Chellappa R (2015) Face-based active authentication on mobile devices. In: 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 1687–1691. IEEE
Galea C, Farrugia RA (2017) Matching software-generated sketches to face photographs with a very deep cnn, morphed faces, and transfer learning. IEEE Trans Inform Forensics Secur 13(6):1421–1431
Galea C, Farrugia RA (2018) Matching software-generated sketches to face photographs with a very deep cnn, morphed faces, and transfer learning. IEEE Trans Inform Forensics Sec 13(6):1421–1431. https://doi.org/10.1109/TIFS.2017.2788002
Grgic M, Delac K, Grgic S (2011) Scface-surveillance cameras face database. Multimed Tools Appl 51(3):863–879
Guo G, Zhang N (2019) A survey on deep learning based face recognition. Comput Vis Image Underst 189:102805
Guo Y, Zhang L, Hu Y, He X, Gao J (2016) Ms-celeb-1m: a dataset and benchmark for large-scale face recognition. CoRR arXiv:1607.08221
Haghighat M, Abdel-Mottaleb M (2017) Low resolution face recognition in surveillance systems using discriminant correlation analysis. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017), pp 912–917. IEEE
Han S, Mao H, Dally WJ (2015) Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149
Harandi MT, Sanderson C, Shirazi S, Lovell BC (2011) Graph embedding discriminant analysis on grassmannian manifolds for improved image set matching. In: CVPR, p 2705–2712
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: International conference on computer vision (ICCV), pp 1389–1397
Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531
Howard A, Sandler M, Chu G, Chen LC, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V, et al (2019) Searching for mobilenetv3. In: Proceedings of the IEEE international conference on computer vision, pp 1314–1324
Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141
Hu Q, Wang P, Cheng J (2018) From hashing to cnns: training binary weight networks via hashing. In: Thirty-Second AAAI conference on artificial intelligence
Huang G, Liu S, Van der Maaten L, Weinberger KQ (2018) Condensenet: An efficient densenet using learned group convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2752–2761
Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708
Huang GB, Ramesh M, Berg T, Learned-miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments
Huang Y, Shen P, Tai Y, Li S, Liu X, Li J, Huang F, Ji R (2020) Improving face recognition from hard samples via distribution distillation loss. arXiv preprint arXiv:2002.03662
Huang Z, Shan S, Wang R, Zhang H, Lao S, Kuerban A, Chen X (2015) A benchmark and comparative study of video-based face recognition on cox face database. IEEE Trans Image Process 24(12):5967–5981
Huang Z, Wang R, Shan S, Chen X (2014) Learning euclidean-to-riemannian metric for point-to-set classification. In: CVPR, p 1677–1684
Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \(< 0.5\ \text{mb}\) model size. arXiv preprint arXiv:1602.07360
IBUG (2019) Lightweight face recognition challenge and workshop (ICCV 2019). https://ibug.doc.ic.ac.uk/. Accessed 2019 May 09
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167
Jain AK, Li SZ (2011) Handbook of face recognition, vol 1, Springer
Kemelmacher-Shlizerman I, Seitz SM, Miller D, Brossard E (2016) The megaface benchmark: 1 million faces for recognition at scale. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4873–4882
Lebedev V, Ganin Y, Rakhuba M, Oseledets I, Lempitsky V (2014) Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553
Li F, Zhang B, Liu B (2016) Ternary weight networks. arXiv preprint arXiv:1605.04711
Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2016) Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710
Li X, Wang F, Hu Q, Leng C (2019) Airface:lightweight and efficient model for face recognition. In: The IEEE international conference on computer vision (ICCV) workshops
Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:1312.4400
Liu H, Simonyan K, Yang Y (2018) DARTS: differentiable architecture search. CoRR arXiv:1806.09055
Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) Sphereface: Deep hypersphere embedding for face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 212–220
Liu Y, Peng B, Shi P, Yan H, Zhou Y, Han B, Zheng Y, Lin C, Jiang J, Fan Y, et al (2018) iqiyi-vid: a large dataset for multi-modal person identification. arXiv preprint arXiv:1811.07548
Liu Z, Li J, Shen Z, Huang G, Yan S, Zhang C (2017) Learning efficient convolutional networks through network slimming. In: Proceedings of the IEEE international conference on computer vision, pp 2736–2744
Llanes NM, Castillo-Rosado K, Méndez-Vázquez H, Khellat-Kihel S, Tistarelli M (2019) Face recognition on mobile devices based on frames selection. In: Progress in pattern recognition, image analysis, computer vision, and applications - 24th Iberoamerican congress, CIARP 2019, Havana, Cuba, October 28–31, 2019, Proceedings, pp 316–325
Lu Z, Jiang X, Kot A (2018) Deep coupled resnet for low-resolution face recognition. IEEE Signal Process Lett 25(4):526–530
Ma N, Zhang X, Zheng HT, Sun J (2018) Shufflenet v2: practical guidelines for efficient cnn architecture design. arXiv preprint arXiv:1807.11164
Mandal B, Lim RY, Dai P, Sayed MR, Li L, Lim JH (2016) Trends in machine and human face recognition. In: Advances in face detection and facial image analysis, pp 145–187. Springer
Martindez-Diaz Y, Luevano LS, Mendez-Vazquez H, Nicolas-Diaz M, Chang L, Gonzalez-Mendoza M (2019) Shufflefacenet: a lightweight face architecture for efficient and highly-accurate face recognition. In: The IEEE international conference on computer vision (ICCV) workshops
Martínez-Díaz Y, Chang L, Hernández N, Méndez-Vázquez H, Sucar LE (2016) Efficient video face recognition by using fisher vector encoding of binary features. In: ICPR, pp 1436–1441
Martínez-Díaz Y, Hernandez N, Biscay RJ, Chang L, Mendez-Vazquez H, Sucar LE (2018) On fisher vector encoding of binary features for video face recognition. J Vis Commun Image Represent 51:155–161
Martínez-Díaz Y, Méndez-Vázquez H, López-Avila L, Chang L, Enrique Sucar L, Tistarelli M (2018) Toward more realistic face recognition evaluation protocols for the youtube faces database. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 413–421
Maze B, Adams J, Duncan JA, Kalka N, Miller T, Otto C, Jain AK, Niggel WT, Anderson J, Cheney J, et al (2018) Iarpa janus benchmark-c: Face dataset and protocol. In: 2018 International conference on biometrics (ICB), pp 158–165. IEEE
Moschoglou S, Papaioannou A, Sagonas C, Deng J, Kotsia I, Zafeiriou S (2017) Agedb: the first manually collected, in-the-wild age database. pp 1997–2005. https://doi.org/10.1109/CVPRW.2017.250
Méndez-Vázquez H, Becerra-Riera F, Morales-González A, López-Avila L, Tistarelli M (2019) Local deep features for composite face sketch recognition. In: 2019 7th international workshop on biometrics and forensics (IWBF), pp 1–6. https://doi.org/10.1109/IWBF.2019.8739212
Ng HW, Winkler S (2014) A data-driven approach to cleaning large face datasets. In: 2014 IEEE international conference on image processing (ICIP), pp 343–347. IEEE
Park W, Kim D, Lu Y, Cho M (2019) Relational knowledge distillation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3967–3976
Parkhi OM, Simonyan K, Vedaldi A, Zisserman A (2014) A compact and discriminative face track descriptor. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1693–1700
Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. Br Mach Vis Conf 1:1–12
Rastegari M, Ordonez V, Redmon J, Farhadi A (2016) Xnor-net: imagenet classification using binary convolutional neural networks. In: European conference on computer vision, pp 525–542. Springer
Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550
Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520
Schroff F, Kalenichenko D, Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 815–823
Sengupta S, Chen J, Castillo C, Patel VM, Chellappa R, Jacobs DW (2016) Frontal to profile face verification in the wild. In: 2016 IEEE winter conference on applications of computer vision (WACV), pp 1–9. https://doi.org/10.1109/WACV.2016.7477558
Sepas-Moghaddam A, Pereira F, Correia PL (2019) Face recognition: a novel multi-level taxonomy based survey. arXiv preprint arXiv:1901.00713
Sun Y, Liang D, Wang X, Tang X (2015) Deepid3: Face recognition with very deep neural networks. arXiv preprint arXiv:1502.00873
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9
Trillionpairs (2019) http://trillionpairs.deepglint.com/overview. Accessed: 2020-07-23
Taigman Y, Yang M, Ranzato M, Wolf L (2014) Closing the gap to human-level performance in face verification. Deepface. IEEE Comput Vis Pattern Recognit (CVPR) 5:6
Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2820–2828
Tan M, Le QV (2019) Efficientnet: rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946
Thomee B, Shamma DA, Friedland G, Elizalde B, Ni K, Poland D, Borth D, Li LJ (2015) Yfcc100m: the new data in multimedia research. arXiv preprint arXiv:1503.01817
Tran LQ, Yin X, Liu X (2018) Representation learning by rotating your faces. IEEE Trans Pattern Anal Mach Intell 41:3007–3021
Tung F, Mori G (2019) Similarity-preserving knowledge distillation. In: Proceedings of the IEEE international conference on computer vision, pp 1365–1374
Turk M, Pentland A (1991) Eigenfaces for recognition. J Cognit Neurosci 3(1):71–86
Wang F, Cheng J, Liu W, Liu H (2018) Additive margin softmax for face verification. IEEE Signal Process Lett 25(7):926–930
Wang M, Deng W (2018) Deep face recognition: a survey. arXiv preprint arXiv:1804.06655
Wang P, Cheng J (2016) Accelerating convolutional neural networks for mobile applications. In: Proceedings of the 24th ACM international conference on multimedia, pp 541–545
Wang Q, Guo G (2019) Benchmarking deep learning techniques for face recognition. J Vis Commun Image Represent 65:102,663
Wang R, Guo H, Davis LS, Dai Q (2012) Covariance discriminative learning: a natural and efficient approach to image set classification. In: CVPR, p 2496–2503
Wang X, Wang S, Zhang S, Fu T, Shi H, Mei T (2018) Support vector guided softmax loss for face recognition. arXiv preprint arXiv:1812.11317
Wen Y, Zhang K, Li Z, Qiao Y (2016) A discriminative feature learning approach for deep face recognition. In: European conference on computer vision, pp 499–515. Springer
Whitelam C, Taborsky E, Blanton A, Maze B, Adams J, Miller T, Kalka N, Jain AK, Duncan JA, Allen K, et al (2017) Iarpa janus benchmark-b face dataset. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 90–98
Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: CVPR, pp 529–534
Wu B, Wan A, Yue X, Jin P, Zhao S, Golmant N, Gholaminejad A, Gonzalez J, Keutzer K (2017) Shift: A zero flop, zero parameter alternative to spatial convolutions. arXiv preprint arXiv:1711.08141
Wu X, He R, Sun Z, Tan T (2018) A light cnn for deep face representation with noisy labels. IEEE Trans Inform Forensics Secur 13(11):2884–2896
Xie S, Zheng H, Liu C, Lin L (2018) SNAS: stochastic neural architecture search. CoRR arXiv:1812.09926
Xie W, Shen L, Zisserman A (2018) Comparator networks. In: Proceedings of the European conference on computer vision (ECCV), pp 782–797
Xie W, Zisserman A (2018) Multicolumn networks for face recognition. arXiv preprint arXiv:1807.09192
Yan M, Zhao M, Xu Z, Zhang Q, Wang G, Su Z (2019) Vargfacenet: An efficient variable group convolutional neural network for lightweight face recognition. In: The IEEE international conference on computer vision (ICCV) workshops
Yang F, Yang W, Gao R, Liao Q (2017) Discriminative multidimensional scaling for low-resolution face recognition. IEEE Signal Process Lett 25(3):388–392
Yang J, Ren P, Chen D, Wen F, Li H, Hua G (2017) Neural aggregation network for video face recognition. In: IEEE conference on computer vision and pattern recognition (CVPR), In press
Yin X, Tai Y, Huang Y, Liu X (2019) Fan: feature adaptation network for surveillance face recognition and normalization. arXiv preprint arXiv:1911.11680
Zagoruyko S, Komodakis N (2016) Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928
Zeng D, Chen H, Zhao Q (2016) Towards resolution invariant face recognition in uncontrolled scenarios. In: 2016 International conference on biometrics (ICB), pp 1–8. IEEE
Zhang J (2019) Seesaw-net: convolution neural network with uneven group convolution. arXiv preprint arXiv:1905.03672
Zhang J (2019) Seesawfacenets: sparse and robust face verification model for mobile platform. arXiv preprint arXiv:1908.09124
Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503
Zhang Q, Li J, Yao M, Song L, Zhou H, Li Z, Meng W, Zhang X, Wang G (2019) Vargnet: variable group convolutional neural network for efficient embedded computing. arXiv preprint arXiv:1907.05653
Zhang X, Zhou X, Lin M, Sun J (2017) Shufflenet: an extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083
Zhang X, Zou J, He K, Sun J (2015) Accelerating very deep convolutional networks for classification and detection. IEEE Trans Pattern Anal Mach Intell 38(10):1943–1955
Zheng T, Deng W (2018) Cross-pose lfw: a database for studying crosspose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Technical report, pp 18–01
Zheng T, Deng W, Hu J (2017) Cross-age lfw: a database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197
Zhi-Peng F, Yan-Ning Z, Hai-Yan H (2014) Survey of deep learning in face recognition. In: 2014 International conference on orange technologies, pp 5–8. IEEE
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Martínez-Díaz, Y., Nicolás-Díaz, M., Méndez-Vázquez, H. et al. Benchmarking lightweight face architectures on specific face recognition scenarios. Artif Intell Rev 54, 6201–6244 (2021). https://doi.org/10.1007/s10462-021-09974-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-021-09974-2