Skip to main content
SpringerLink
Log in

Kinship verification from facial images and videos: human versus machine

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Automatic kinship verification from facial images is a relatively new and challenging research problem in computer vision. It consists in automatically determining whether two persons have a biological kin relation by examining their facial attributes. In this work, we compare the performance of humans and machines in kinship verification tasks. We investigate the state-of-the-art methods in automatic kinship verification from facial images, comparing their performance with the one obtained by asking humans to complete an equivalent task using a crowdsourcing system. Our results show that machines can consistently beat humans in kinship classification tasks in both images and videos. In addition, we study the limitations of currently available kinship databases and analyzing their possible impact in kinship verification experiment and this type of comparison.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)

    Article  MATH  Google Scholar 

  2. Ahonen, T., Rahtu, E., Ojansivu, V., Heikkila, J.: Recognition of blurred faces using local phase quantization. In: International Conference on Pattern Recognition, pp. 1–4 (2008)

  3. Alirezazadeh, P., Fathi, A., Abdali-Mohammadi, F.: A genetic algorithm-based feature selection for kinship verification. Signal Process. Lett. IEEE 22(12), 2459–2463 (2015). https://doi.org/10.1109/LSP.2015.2490805

    Article  Google Scholar 

  4. Alvergne, A., Faurie, C., Raymond, M.: Differential facial resemblance of young children to their parents: who do children look like more? Evol. Hum. Behav. 28(2), 135–144 (2007)

    Article  Google Scholar 

  5. Alvergne, A., Oda, R., Faurie, C., Matsumoto-Oda, A., Durand, V., Raymond, M.: Cross-cultural perceptions of facial resemblance between kin. J. Vis. 9(6), 23–23 (2009)

    Article  Google Scholar 

  6. Alvergne, A., Perreau, F., Mazur, A., Mueller, U., Raymond, M.: Identification of visual paternity cues in humans. Biol. Lett. 10(4), 20140063 (2014)

    Article  Google Scholar 

  7. Amazon: Amazon mechanical turk. (2016). https://www.mturk.com/

  8. Anastasi, J.S., Rhodes, M.G.: An own-age bias in face recognition for children and older adults. Psychon. Bull. Rev. 12(6), 1043–1047 (2005). https://doi.org/10.3758/BF03206441

    Article  Google Scholar 

  9. Barkan, O., Weill, J., Wolf, L., Aronowitz, H.: Fast high dimensional vector multiplication face recognition. In: IEEE International Conference on Computer Vision, pp. 1960–1967 (2013)

  10. Bordallo López, M., Boutellaa, E., Hadid, A.: Comments on the “kinship face in the wild” data sets. IEEE Trans. Pattern Anal. Mach. Intell. PP(99), 1–1 (2016). https://doi.org/10.1109/TPAMI.2016.2522416

    Google Scholar 

  11. Bottino, A.G., De Simone, M., Laurentini, A., Vieira, T.: A new problem in face image analysis: finding kinship clues for siblings pairs (2012)

  12. Bottino, A.G., Ul Islam, I., Vieira, T.: A multi-perspective holistic approach to kinship verification in the wild (2015)

  13. Boutellaa, E., López, B., M. Ait-Aoudia, S., Feng, X., Hadid, A.: Kinship verification from videos using texture spatio-temporal features and deep learning features. In: International Conference on Biometrics (ICB’16) (2016)

  14. Brédart, S., French, R.M.: Do babies resemble their fathers more than their mothers? a failure to replicate christenfeld and hill (1995). Evol. Hum. Behav. 20(2), 129–135 (1999)

    Article  Google Scholar 

  15. Bressan, P., Grassi, M.: Parental resemblance in 1-year-olds and the gaussian curve. Evol. Hum. Behav. 25(3), 133–141 (2004)

    Article  Google Scholar 

  16. Burch, R.L., Gallup, G.G.: Perceptions of paternal resemblance predict family violence. Evol. Hum. Behav. 21(6), 429–435 (2000)

    Article  Google Scholar 

  17. Chan, C.H., Tahir, M., Kittler, J., Pietikainen, M.: Multiscale local phase quantization for robust component-based face recognition using kernel fusion of multiple descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1164–1177 (2013)

    Article  Google Scholar 

  18. Chen, J., Shan, S., He, C., Zhao, G., Pietikainen, M., Chen, X., Gao, W.: Wld: a robust local image descriptor. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1705–1720 (2010)

    Article  Google Scholar 

  19. Chen, X., An, L., Yang, S., Wu, W.: Kinship verification in multi-linear coherent spaces. Multimed. Tools Appl. 76, 1–18 (2015)

    Google Scholar 

  20. Christenfeld, N., Hill, E.A., et al.: Whose baby are you. Nature 378(6558), 669–669 (1995)

    Article  Google Scholar 

  21. Dal Martello, M.F., Maloney, L.T.: Where are kin recognition signals in the human face? J. Vis. 6(12), 2–2 (2006)

    Article  Google Scholar 

  22. Dal Martello, M.F., Maloney, L.T.: Lateralization of kin recognition signals in the human face. J. Vis. 10(8), 9 (2010)

    Article  Google Scholar 

  23. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005)

  24. DeBruine, L.M., Smith, F.G., Jones, B.C., Roberts, S.C., Petrie, M., Spector, T.D.: Kin recognition signals in adult faces. Vis. Res. 49(1), 38–43 (2009)

    Article  Google Scholar 

  25. Dehghan, A., Ortiz, E., Villegas, R., Shah, M.: Who do i look like? determining parent-offspring resemblance via gated autoencoders. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1757–1764 (2014). https://doi.org/10.1109/CVPR.2014.227

  26. Dibeklioglu, H.: Visual transformation aided contrastive learning for video-based kinship verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2459–2468 (2017)

  27. Dibeklioglu, H., Salah, A., Gevers, T.: Are you really smiling at me? spontaneous versus posed enjoyment smiles. In: Computer Vision ECCV 2012, vol. 7574, pp. 525–538. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-33712-3_38

  28. Dibeklioglu, H., Salah, A., Gevers, T.: Like father, like son: Facial expression dynamics for kinship verification. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1497–1504 (2013)

  29. Downs, J.S., Holbrook, M.B., Sheng, S., Cranor, L.F.: Are your participants gaming the system?: screening mechanical turk workers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2399–2402. ACM (2010)

  30. Fang, R., Tang, K.D., Snavely, N., Chen, T.: Towards computational models of kinship verification. In: 2010 17th IEEE International Conference on Image Processing (ICIP), pp. 1577–1580. IEEE (2010)

  31. Ghahramani, M., Yau, W.Y., Teoh, E.K.: Family verification based on similarity of individual family member’s facial segments. Mach. Vis. Appl. 25(4), 919–930 (2014)

    Article  Google Scholar 

  32. Goncalves, J., Hosio, S., Rogstadius, J., Karapanos, E., Kostakos, V.: Motivating participation and improving quality of contribution in ubiquitous crowdsourcing. Comput. Netw. 90, 34–48 (2015)

    Article  Google Scholar 

  33. Guo, G., Wang, X.: Kinship measurement on salient facial features. IEEE Trans. Instrum. Meas. 61(8), 2322–2325 (2012). https://doi.org/10.1109/TIM.2012.2187468

    Article  Google Scholar 

  34. Hamilton, W.: The genetical evolution of social behaviour (1964)

  35. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, pp. 630–645. Springer, Berlin (2016)

  36. Hills, P.J., Lewis, M.B.: The own-age face recognition bias in children and adults. Q. J. Exp. Psychol. 64(1), 17–23 (2011)

    Article  Google Scholar 

  37. Hu, J., Lu, J., Tan, Y.P., Yuan, J., Zhou, J.: Local large-margin multi-metric learning for face and kinship verification. IEEE Trans. Circuits Syst. Video Technol. (2017)

  38. Hu, J., Lu, J., Yuan, J., Tan, Y.P.: Large margin multi-metric learning for face and kinship verification in the wild. In: Computer Vision–ACCV 2014, pp. 252–267. Springer, Berlin (2015)

  39. Kaminski, G., Gentaz, E., Mazens, K.: Development of children’s ability to detect kinship through facial resemblance. Anim. Cognit. 15(3), 421–427 (2012)

    Article  Google Scholar 

  40. Kaminski, G., Méary, D., Mermillod, M., Gentaz, E.: Perceptual factors affecting the ability to assess facial resemblance between parents and newborns in humans. Perception 39(6), 807–818 (2010)

    Article  Google Scholar 

  41. Kan, M., Shan, S., Xu, D., Chen, X.: Side-information based linear discriminant analysis for face recognition. In: British Machine Vision Conference, pp. 1–12 (2011)

  42. Kannala, J., Rahtu, E.: BSIF: Binarized statistical image features. In: International Conference on Pattern Recognition (ICPR), pp. 1363–1366 (2012)

  43. Kohli, N., Singh, R., Vatsa, M.: Self-similarity representation of weber faces for kinship classification. In: 2012 IEEE Fifth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 245–250 (2012)

  44. Kou, L., Zhou, X., Xu, M., Shang, Y.: Learning a genetic measure for kinship verification using facial images. Math. Probl. Eng. 2015 (2015)

  45. Lu, J., Hu, J., Liong, V.E., Zhou, X., Bottino, A., Islam, I.U., Vieira, T.F., Qin, X., Tan, X., Keller, Y., et al.: The FG 2015 kinship verification in the wild evaluation. Face and Gesture (FG’15) (2015)

  46. Lu, J., Hu, J., Tan, Y.P.: Discriminative deep metric learning for face and kinship verification. IEEE Trans. Image Process. 26(9), 4269–4282 (2017)

    Article  MathSciNet  Google Scholar 

  47. Lu, J., Hu, J., Zhou, X., Zhou, J., Castrillón-Santana, M., Lorenzo-Navarro, J., Kou, L., Shang, Y., Bottino, A., Vieira, T.F.: Kinship verification in the wild: The first kinship verification competition. In: 2014 IEEE International Joint Conference on Biometrics (IJCB), pp. 1–6. IEEE (2014)

  48. Lu, J., Zhou, X., Tan, Y.P., Shang, Y., Zhou, J.: Neighborhood repulsed metric learning for kinship verification. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 331–345 (2014)

    Article  Google Scholar 

  49. Maloney, L.T., Dal Martello, M.F.: Kin recognition and the perceived facial similarity of children. J. Vis. 6(10), 4–4 (2006)

    Article  Google Scholar 

  50. Mateo, J.M.: Perspectives: Hamilton’s legacy: mechanisms of kin recognition in humans. Ethology 121(5), 419–427 (2015)

    Article  Google Scholar 

  51. Mathews, S., Kambhamettu, C., Barner, K.: Am I your sibling? inferring kinship cues from facial image pairs. In: 2015 49th Annual Conference on Information Sciences and Systems (CISS), pp. 1–5 (2015). https://doi.org/10.1109/CISS.2015.7086888

  52. McLain, D.K., Setters, D., Moulton, M.P., Pratt, A.E.: Ascription of resemblance of newborns by parents and nonrelatives. Evol. Hum. Behav. 21(1), 11–23 (2000)

    Article  Google Scholar 

  53. Meissner, C.A., Brigham, J.C.: Thirty years of investigating the own-race bias in memory for faces: a meta-analytic review. Psychol. Public Policy Law 7(1), 3 (2001)

    Article  Google Scholar 

  54. Nesse, R.M., Silverman, A., Bortz, A.: Sex differences in ability to recognize family resemblance. Ethol. Sociobiol. 11(1), 11–21 (1990). https://doi.org/10.1016/0162-3095(90)90003-O

    Article  Google Scholar 

  55. Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference (2015)

  56. Peleg, G., Katzir, G., Peleg, O., Kamara, M., Brodsky, L., Hel-Or, H., Keren, D., Nevo, E.: Hereditary family signature of facial expression. Proc. Natl. Acad. Sci. 103(43), 15921–15926 (2006)

    Article  Google Scholar 

  57. Platek, S.M., Burch, R.L., Panyavin, I.S., Wasserman, B.H., Gallup, G.G.: Reactions to children’s faces: resemblance affects males more than females. Evol. Hum. Behav. 23(3), 159–166 (2002)

    Article  Google Scholar 

  58. Platek, S.M., Keenan, J.P., Mohamed, F.B.: Sex differences in the neural correlates of child facial resemblance: an event-related fmri study. NeuroImage 25(4), 1336–1344 (2005)

    Article  Google Scholar 

  59. Platek, S.M., Kemp, S.M.: Is family special to the brain? an event-related fmri study of familiar, familial, and self-face recognition. Neuropsychologia 47(3), 849–858 (2009)

    Article  Google Scholar 

  60. Porter, R.H., Cernoch, J.M., Balogh, R.D.: Recognition of neonates by facial-visual characteristics. Pediatrics 74(4), 501–504 (1984)

    Google Scholar 

  61. Qin, X., Tan, X., Chen, S.: Tri-subject kinship verification: understanding the core of a family. IEEE Trans. Multimed. 17, 1855–1867 (2015)

    Article  Google Scholar 

  62. Robinson, J.P., Shao, M., Wu, Y., Fu, Y.: Families in the wild (fiw): large-scale kinship image database and benchmarks. In: Proceedings of the 2016 ACM on Multimedia Conference, MM ’16, pp. 242–246. ACM, New York, NY, USA (2016)

  63. Robinson, J.P., Shao, M., Zhao, H., Wu, Y., Gillis, T., Fu, Y.: Recognizing families in the wild (rfiw): data challenge workshop in conjunction with acm mm 2017. In: Proceedings of the 2017 Workshop on Recognizing Families In the Wild, RFIW ’17, pp. 5–12. ACM, New York, NY, USA (2017)

  64. Simonyan, K., Parkhi, O., Vedaldi, A., Zisserman, A.: Fisher vector faces in the wild. In: Proceedings of the British Machine Vision Conference. BMVA Press (2013)

  65. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014). http://arxiv.org/abs/1409.1556

  66. Somanath, G., Kambhamettu, C.: Can faces verify blood-relations? In: International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 105–112. IEEE (2012)

  67. Sun, Y., Wang, X., Tang, X.: Deep learning face representation from predicting 10,000 classes. In: Computer Vision and Pattern Recognition, pp. 1891–1898. IEEE, Washington, DC, USA (2014). https://doi.org/10.1109/CVPR.2014.244

  68. Wang, S., Robinson, J.P., Fu, Y.: Kinship verification on families in the wild with marginalized denoising metric learning. In: 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017), pp. 216–221 (2017)

  69. Wang, X., Kambhamettu, C.: Leveraging appearance and geometry for kinship verification. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 5017–5021. IEEE (2014)

  70. Wolf, L., Hassner, T., Taigman, Y.: Descriptor based methods in the wild. In: Real-Life Images workshop at the European Conference on Computer Vision (2008)

  71. Wu, H., Yang, S., Sun, S., Liu, C., Luo, Y.J.: The male advantage in child facial resemblance detection: behavioral and erp evidence. Soc. Neurosci. 8(6), 555–567 (2013)

    Article  Google Scholar 

  72. Wu, X., Boutellaa, E., Lpez, M.B., Feng, X., Hadid, A.: On the usefulness of color for kinship verification from face images. In: 2016 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–6 (2016)

  73. Xia, S., Shao, M., Fu, Y.: Toward kinship verification using visual attributes. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp. 549–552. IEEE (2012)

  74. Xia, S., Shao, M., Luo, J., Fu, Y.: Understanding kin relationships in a photo. IEEE Trans. Multimed. 14(4), 1046–1056 (2012)

    Article  Google Scholar 

  75. Yan, H., Hu, J.: Video-based kinship verification using distance metric learning. Pattern Recognit. 75, 15–24 (2018)

    Article  Google Scholar 

  76. Yan, H., Lu, J.: Video-based facial kinship verification. In: Facial Kinship Verification, pp. 63–80. Springer, Singapore (2017)

  77. Yan, H., Lu, J., Deng, W., Zhou, X.: Discriminative multimetric learning for kinship verification. IEEE Trans. Inf. Forensics Secur. 9(7), 1169–1178 (2014)

    Article  Google Scholar 

  78. Yan, H.C., Lu, J., Zhou, X.: Prototype-based discriminative feature learning for kinship verification (2014)

  79. Yi, D., Lei, Z., Liao, S., Li, S.Z.: Learning face representation from scratch. arXiv:1411.7923 (2014)

  80. Zhang, J., Xia, S., Pan, H., Qin, A.: A genetics-motivated unsupervised model for tri-subject kinship verification. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2916–2920. IEEE (2016)

  81. Zhang, K., Huang, Y., Song, C., Wu, H., Wang, L.: Kinship verification with deep convolutional neural networks. In: British Machine Vision Conference (BMVC), pp. 148.1–148.12 (2015)

  82. Zheng, L., Idrissi, K., Garcia, C., Duffner, S., Baskurt, A. (eds.): Triangular similarity metric learning for face verification. IEEE (2015)

  83. Zhou, X., Hu, J., Lu, J., Shang, Y., Guan, Y.: Kinship verification from facial images under uncontrolled conditions. In: International Conference on Multimedia, MM ’11, pp. 953–956. ACM, New York, NY, USA (2011)

  84. Zhou, X., Lu, J., Hu, J., Shang, Y.: Gabor-based gradient orientation pyramid for kinship verification under uncontrolled environments. In: Proceedings of the 20th ACM International Conference on Multimedia, pp. 725–728 (2012)

  85. Zhou, X., Shang, Y., Yan, H., Guo, G.: Ensemble similarity learning for kinship verification from facial images in the wild. Inf. Fus. 32, 40–48 (2015)

    Article  Google Scholar 

Download references

Acknowledgements

The support of the Academy of Finland is fully acknowledged.

Author information

Authors and Affiliations

  1. Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland

    Miguel Bordallo Lopez, Abdenour Hadid & Elhocine Boutellaa

  2. School of Computing and Information Systems, University of Melbourne, Melbourne, Australia

    Jorge Goncalves & Vassilis Kostakos

  3. Center for Ubiquitous Computing, University of Oulu, Oulu, Finland

    Simo Hosio

Authors
  1. Miguel Bordallo Lopez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdenour Hadid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elhocine Boutellaa
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jorge Goncalves
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vassilis Kostakos
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Simo Hosio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel Bordallo Lopez.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 287 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bordallo Lopez, M., Hadid, A., Boutellaa, E. et al. Kinship verification from facial images and videos: human versus machine. Machine Vision and Applications 29, 873–890 (2018). https://doi.org/10.1007/s00138-018-0943-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-018-0943-x

Keywords

Search

Navigation