Skip to main content
SpringerLink
Log in

RepMix: Representation Mixing for Robust Attribution of Synthesized Images

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13674))

Included in the following conference series:

Abstract

Rapid advances in Generative Adversarial Networks (GANs) raise new challenges for image attribution; detecting whether an image is synthetic and, if so, determining which GAN architecture created it. Uniquely, we present a solution to this task capable of 1) matching images invariant to their semantic content; 2) robust to benign transformations (changes in quality, resolution, shape, etc.) commonly encountered as images are re-shared online. In order to formalize our research, a challenging benchmark, Attribution88, is collected for robust and practical image attribution. We then propose RepMix, our GAN fingerprinting technique based on representation mixing and a novel loss. We validate its capability of tracing the provenance of GAN-generated images invariant to the semantic content of the image and also robust to perturbations. We show our approach improves significantly from existing GAN fingerprinting works on both semantic generalization and robustness. Data and code are available at https://github.com/TuBui/image_attribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://en.wikipedia.org/wiki/Beta_distribution.

References

  1. Ashual, O., Wolf, L.: Specifying object attributes and relations in interactive scene generation. In: Proceedings of the ICCV, pp. 4561–4569 (2019)

    Google Scholar 

  2. Asnani, V., Yin, X., Hassner, T., Liu, X.: Reverse engineering of generative models: Inferring model hyperparameters from generated images. arXiv preprint arXiv:2106.07873 (2021)

  3. Aythora, J., et al.: Multi-stakeholder media provenance management to counter synthetic media risks in news publishing. In: Proceedings of the International Broadcasting Convention (IBC) (2020)

    Google Scholar 

  4. Baba, S., Krekor, L., Arif, T., Shaaban, Z.: Watermarking scheme for copyright protection of digital images. IJCSNS 9(4) (2019)

    Google Scholar 

  5. Bellemare, M.G., et al.: The cramer distance as a solution to biased wasserstein gradients. arXiv preprint arXiv:1705.10743 (2017)

  6. Bińkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying mmd gans. In: Proceedings of the ICLR (2018)

    Google Scholar 

  7. Black, A., Bui, T., Jenni, S., Swaminathan, V., Collomosse, J.: Vpn: Video provenance network for robust content attribution. In: Proceedings of the CVMP, pp. 1–10 (2021)

    Google Scholar 

  8. Black, A., Bui, T., Jin, H., Swaminathan, V., Collomosse, J.: Deep image comparator: learning to visualize editorial change. In: Proceedings of the CVPR, pp. 972–980 (2021)

    Google Scholar 

  9. Bui, T., et al.: Archangel: tamper-proofing video archives using temporal content hashes on the blockchain. In: Proceedings of the CVPR WS (2019)

    Google Scholar 

  10. Bui, T., et al.: Tamper-proofing video with hierarchical attention autoencoder hashing on blockchain. IEEE Trans. Multimedia 22(11), 2858–2872 (2020)

    Article  Google Scholar 

  11. (CAI), C.A.I.: Setting the standard for content attribution. Technical report, Adobe Inc. (2020)

    Google Scholar 

  12. Cao, Z., Long, M., Wang, J., Yu, P.S.: Hashnet: deep learning to hash by continuation. In: Proceedings of the CVPR, pp. 5608–5617 (2017)

    Google Scholar 

  13. Chen, A., Liu, R., Xie, L., Chen, Z., Su, H., Yu, J.: Sofgan: a portrait image generator with dynamic styling. ACM Trans. Graphics (TOG) 41(1), 1–26 (2022)

    Google Scholar 

  14. Devi, P., Venkatesan, M., Duraiswamy, K.: A fragile watermarking scheme for image authentication with tamper localization using integer wavelet transform. J. Comput. Sci. 5(11), 831–837 (2019)

    Google Scholar 

  15. Ding, Y., Thakur, N., Li, B.: Does a gan leave distinct model-specific fingerprints? In: Proceedings of the BMVC (2021)

    Google Scholar 

  16. Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., Ferrer, C.C.: The deepfake detection challenge (DFDC) dataset. CoRR abs/2006.07397 (2020), arxiv.org/abs/2006.07397

  17. Frank, J., Eisenhofer, T., Schönherr, L., Fischer, A., Kolossa, D., Holz, T.: Leveraging frequency analysis for deep fake image recognition. In: Proceedings of the ICML, pp. 3247–3258. PMLR (2020)

    Google Scholar 

  18. Girish, S., Suri, S., Rambhatla, S.S., Shrivastava, A.: Towards discovery and attribution of open-world gan generated images. In: Proceedings of the ICCV, pp. 14094–14103 (2021)

    Google Scholar 

  19. Goodfellow, I., et al.: Generative adversarial nets. NeurIPS 27 (2014)

    Google Scholar 

  20. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)

  21. Hameed, K., Mumtax, A., Gilani, S.: Digital image watermarking in the wavelet transform domain. WASET 13, 86–89 (2006)

    Google Scholar 

  22. He, Y., Yu, N., Keuper, M., Fritz, M.: Beyond the spectrum: Detecting deepfakes via re-synthesis. In: Proceedings of the IJCAI-21, pp. 2534–2541. International Joint Conferences on Artificial Intelligence Organization (2021)

    Google Scholar 

  23. Hendrycks, D., et al.: The many faces of robustness: a critical analysis of out-of-distribution generalization. In: Proceedings of the ICCV, pp. 8340–8349 (2021)

    Google Scholar 

  24. Hendrycks, D., Dietterich, T.: Benchmarking neural network robustness to common corruptions and perturbations. In: Proceedings of the ICLR (2018)

    Google Scholar 

  25. Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: Augmix: a simple data processing method to improve robustness and uncertainty. In: Proceedings of the ICLR (2019)

    Google Scholar 

  26. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of gans for improved quality, stability, and variation. In: Proceedings of the ICLR (2018)

    Google Scholar 

  27. Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., Aila, T.: Alias-free generative adversarial networks. NeurIPS 34 (2021)

    Google Scholar 

  28. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the CVPR, pp. 4401–4410 (2019)

    Google Scholar 

  29. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the CVPR, pp. 8110–8119 (2020)

    Google Scholar 

  30. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. NeurIPS 25 (2012)

    Google Scholar 

  31. Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the CVPR, pp. 4681–4690 (2017)

    Google Scholar 

  32. Li, W., Wang, S., Kang, W.C.: Feature learning based deep supervised hashing with pairwise labels. In: Proceedings of the IJCAI, pp. 1711–1717 (2016)

    Google Scholar 

  33. Li, Y., Ching, M.C., Lyu, S.: In ictu oculi: Exposing ai created fake videos by detecting eye blinking. In: Proceedings of the IEEE WIFS (2018)

    Google Scholar 

  34. Liu, H., Wang, R., Shan, S., Chen, X.: Deep supervised hashing for fast image retrieval. In: Proceedings of the CVPR, pp. 2064–2072 (2016)

    Google Scholar 

  35. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the ICCV, pp. 3730–3738 (2015)

    Google Scholar 

  36. Lu, Y., Wu, S., Tai, Y.-W., Tang, C.-K.: Image generation from sketch constraint using contextual GAN. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 213–228. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_13

    Chapter  Google Scholar 

  37. Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9(11) (2008)

    Google Scholar 

  38. Marra, F., Gragnaniello, D., Verdoliva, L., Poggi, G.: Do gans leave artificial fingerprints? In: Proceedings of the MIPR, pp. 506–511. IEEE (2019)

    Google Scholar 

  39. Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. In: International Conference on Learning Representations (2018)

    Google Scholar 

  40. Nguyen, E., Bui, T., Swaminathan, V., Collomosse, J.: Oscar-net: object-centric scene graph attention for image attribution. In: Proceedings of the ICCV, pp. 14499–14508 (2021)

    Google Scholar 

  41. Nightingale, S.J., Farid, H.: Ai-synthesized faces are indistinguishable from real faces and more trustworthy. Proc. Natl. Acad. Sci. 119(8) (2022)

    Google Scholar 

  42. Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the CVPR, pp. 2337–2346 (2019)

    Google Scholar 

  43. Perarnau, G., Van De Weijer, J., Raducanu, B., Álvarez, J.M.: Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355 (2016)

  44. Profrock, D., Schlauweg, M., Muller, E.: Content-based watermarking by geometric wrapping and feature- based image segmentation. In: Proceedings of the SITIS, pp. 572–581 (2006)

    Google Scholar 

  45. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the ICCV, pp. 618–626 (2017)

    Google Scholar 

  46. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of the ICLR (2015)

    Google Scholar 

  47. Sirovich, L., Kirby, M.: Low-dimensional procedure for the characterization of human faces. Josa a 4(3), 519–524 (1987)

    Article  Google Scholar 

  48. Wan, C.H., Chuang, S.P., Lee, H.Y.: Towards audio to scene image synthesis using generative adversarial network. In: Proceedings of the ICASSP, pp. 496–500. IEEE (2019)

    Google Scholar 

  49. Wang, J., et al.: Generalizing to unseen domains: a survey on domain generalization. IEEE Trans. Knowl. Data Eng. (2022)

    Google Scholar 

  50. Wang, R., et al.: Fakespotter: a simple yet robust baseline for spotting ai-synthesized fake faces. In: Proceedings of the IJCAI, pp. 3444–3451 (2021)

    Google Scholar 

  51. Wang, S.Y., Wang, O., Zhang, R., Owens, A., Efros, A.A.: Cnn-generated images are surprisingly easy to spot... for now. In: Proceedings of the CVPR, pp. 8695–8704 (2020)

    Google Scholar 

  52. Wu, Y., AbdAlmageed, W., Natarajan, P.: Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features. In: Proceedings of theCVPR, pp. 9543–9552 (2019)

    Google Scholar 

  53. Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., Xiao, J.: Lsun: construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015)

  54. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the CVPR, pp. 5505–5514 (2018)

    Google Scholar 

  55. Yu, N., Barnes, C., Shechtman, E., Amirghodsi, S., Lukac, M.: Texture mixer: A network for controllable synthesis and interpolation of texture. In: Proceedings of the CVPR, pp. 12164–12173 (2019)

    Google Scholar 

  56. Yu, N., Davis, L.S., Fritz, M.: Attributing fake images to gans: Learning and analyzing gan fingerprints. In: Proc. ICCV. pp. 7556–7566 (2019)

    Google Scholar 

  57. Yu, N., Skripniuk, V., Abdelnabi, S., Fritz, M.: Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In: Proceedings of the ICCV, pp. 14448–14457 (2021)

    Google Scholar 

  58. Yu, N., Skripniuk, V., Chen, D., Davis, L., Fritz, M.: Responsible disclosure of generative models using scalable fingerprinting (2022)

    Google Scholar 

  59. Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the ICCV, pp. 6023–6032 (2019)

    Google Scholar 

  60. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: beyond empirical risk minimization. In: Proceedings of the ICLR (2018)

    Google Scholar 

  61. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the CVPR, pp. 586–595 (2018)

    Google Scholar 

  62. Zhang, X., Sun, Z.H., Karaman, S., Chang, S.: Discovering image manipulation history by pairwise relation and forensics tools. IEEE J. Selected Topics Signal Process. 14(5), 1012–1023 (2020)

    Google Scholar 

  63. Zhang, X., Karaman, S., Chang, S.F.: Detecting and simulating artifacts in gan fake images. In: IEEE WIFS, pp. 1–6. IEEE (2019)

    Google Scholar 

  64. Zhu, H., Long, M., Wang, J., Cao, Y.: Deep hashing network for efficient similarity retrieval. In: Proceedings of the AAAI (2016)

    Google Scholar 

Download references

Acknowledgments

This work was supported by EPSRC DECaDE Grant Ref EP/T022485/1.

Author information

Authors and Affiliations

  1. University of Surrey, Guildford, UK

    Tu Bui & John Collomosse

  2. Salesforce Research, Palo Alto, USA

    Ning Yu

  3. Adobe Research, San Jose, USA

    John Collomosse

Authors
  1. Tu Bui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ning Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John Collomosse
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tu Bui .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 7586 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bui, T., Yu, N., Collomosse, J. (2022). RepMix: Representation Mixing for Robust Attribution of Synthesized Images. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13674. Springer, Cham. https://doi.org/10.1007/978-3-031-19781-9_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19781-9_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19780-2

  • Online ISBN: 978-3-031-19781-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation