Skip to main content
SpringerLink
Log in

Thinking in Patch: Towards Generalizable Forgery Detection with Patch Transformation

  • Conference paper
  • First Online:
PRICAI 2021: Trends in Artificial Intelligence (PRICAI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13033))

Included in the following conference series:

Abstract

Nowadays, synthetic faces can completely trick human eyes, which raises social concerns for malicious dissemination of such fake content. As a result, face forgery detection has become a significant research topic. Due to the different distributions of synthetic data in different generation algorithms, it is a great challenge to improve the generalization ability of the face forgery detection algorithm. To address this challenge, we propose a general two-stream patch-based face forgery detection network (FDPT), which introduces a patch transformation to encourage the model to focus on stable information in different data. Specifically, a random transformation is designed to help CNN stream extract local subtle artifacts from images. Meanwhile, a sequence transformation is employed to enhance the global spatial representation ability of the image through the CNN-GRU stream. Finally, a fusion strategy is used to improve the detection accuracy. We conduct extensive experiments to show that FDPT achieves state-of-the-art performance on two popular benchmarks. Moreover, FDPT outperforms the recently proposed generalization methods when applied to forgery generated by unseen face manipulation techniques (e.g., 84.39% \(\rightarrow \) 95.53% on Face2Face dataset).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. DeepFakes (2019). https://www.github.com/deepfakes/faceswap

  2. FaceApp (2019). https://faceapp.com/app

  3. FaceSwap (2019). https://www.github.com/MarekKowalski/FaceSwap

  4. Afchar, D., Nozick, V., Yamagishi, J., Echizen, I.: MesoNet: a compact facial video forgery detection network. In: 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–7 (2018)

    Google Scholar 

  5. Bappy, J.H., Simons, C., Nataraj, L., Manjunath, B., Roy-Chowdhury, A.K.: Hybrid LSTM and encoder-decoder architecture for detection of image forgeries. IEEE Trans. Image Process. 28(7), 3286–3300 (2019)

    Article  MathSciNet  Google Scholar 

  6. Berthelot, D., Schumm, T., Metz, L.: BEGAN: Boundary Equilibrium Generative Adversarial Networks. arXiv e-prints arXiv:1703.10717 (2017)

  7. Bianchi, T., De Rosa, A., Piva, A.: Improved DCT coefficient analysis for forgery localization in jpeg images. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2444–2447 (2011)

    Google Scholar 

  8. Chai, L., Bau, D., Lim, S.-N., Isola, P.: What makes fake images detectable? Understanding properties that generalize. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12371, pp. 103–120. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58574-7_7

    Chapter  Google Scholar 

  9. Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8789–8797, June 2018

    Google Scholar 

  10. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1251–1258, July 2017

    Google Scholar 

  11. Cole, S.: AI-assisted fake porn is here and we’re all fucked. Motherboard Tech by Vice, December 2017

    Google Scholar 

  12. Cozzolino, D., Poggi, G., Verdoliva, L.: Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection. In: Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security, pp. 159–164 (2017)

    Google Scholar 

  13. Dang, H., Liu, F., Stehouwer, J., Liu, X., Jain, A.K.: On the detection of digital face manipulation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5781–5790, June 2020

    Google Scholar 

  14. Du, M., Pentyala, S., Li, Y., Hu, X.: Towards generalizable deepfake detection with locality-aware autoencoder. In: Proceedings of the 29th ACM International Conference on Information and Knowledge Management, pp. 325–334 (2020)

    Google Scholar 

  15. Durall, R., Keuper, M., Pfreundt, F.J., Keuper, J.: Unmasking DeepFakes with simple Features. arXiv e-prints arXiv:1911.00686 (2019)

  16. Goodfellow, I., et al.: Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27, 2672–2680 (2014)

    Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778, June 2016

    Google Scholar 

  18. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: Proceedings of International Conference on Learning Representations (ICLR) (2018)

    Google Scholar 

  19. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4401–4410, June 2019

    Google Scholar 

  20. Kingma, D.P., Dhariwal, P.: Glow: generative flow with invertible 1 \(\times \) 1 convolutions. In: Advances in Neural Information Processing Systems NeurIPS 2018, pp. 10236–10245 (2018)

    Google Scholar 

  21. Li, L., et al.: Face x-ray for more general face forgery detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5001–5010, June 2020

    Google Scholar 

  22. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 3730–3738, December 2015

    Google Scholar 

  23. Mayer, O., Stamm, M.C.: Exposing fake images with forensic similarity graphs. IEEE J. Sel. Top. Sig. Process. 14(5), 1049–1064 (2020)

    Article  Google Scholar 

  24. Mayer, O., Stamm, M.C.: Forensic similarity for digital images. IEEE Trans. Inf. Forensics Secur. 15, 1331–1346 (2020)

    Article  Google Scholar 

  25. Nataraj, L., et al.: Detecting GAN generated fake images using co-occurrence matrices. Electron. Imag. 2019(5), 532-1–532-7 (2019)

    Google Scholar 

  26. Qian, Y., Yin, G., Sheng, L., Chen, Z., Shao, J.: Thinking in frequency: face forgery detection by mining frequency-aware clues. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 86–103. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_6

    Chapter  Google Scholar 

  27. Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niessner, M.: Faceforensics++: Learning to detect manipulated facial images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1–11, October 2019

    Google Scholar 

  28. Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: image synthesis using neural textures. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)

    Article  Google Scholar 

  29. Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2387–2395, June 2016

    Google Scholar 

  30. Wang, S.Y., Wang, O., Zhang, R., Owens, A., Efros, A.A.: CNN-generated images are surprisingly easy to spot... for now. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8695–8704, June 2020

    Google Scholar 

  31. Zhang, X., Karaman, S., Chang, S.F.: Detecting and simulating artifacts in GAN fake images. In: 2019 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–6 (2019)

    Google Scholar 

  32. Zhou, P., Han, X., Morariu, V.I., Davis, L.S.: Two-stream neural networks for tampered face detection. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1831–1839 (2017)

    Google Scholar 

Download references

Acknowledgments

This work is supported in part by the Natural Science Foundation of China (NSFC) under Grant U19B2036.

Author information

Authors and Affiliations

  1. University of Science and Technology of China, Hefei, 230026, China

    Xueqi Zhang, Shuo Wang, Min Zhang & Haiyong Xie

  2. Key Laboratory of Cyberculture Content Cognition and Detection, Ministry of Culture and Tourism, Anhui, 230027, China

    Xueqi Zhang, Min Zhang & Haiyong Xie

  3. National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data, Beijing, 100040, China

    Chenyu Liu & Xiaohan Liu

Authors
  1. Xueqi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Min Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaohan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Haiyong Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xueqi Zhang or Haiyong Xie .

Editor information

Editors and Affiliations

  1. MIMOS Berhad, Kuala Lumpur, Malaysia

    Duc Nghia Pham

  2. Sirindhorn International Institute of Science and Technology, Thammasat University, Mueang Pathum Thani, Thailand

    Thanaruk Theeramunkong

  3. Data61, CSIRO, Brisbane, QLD, Australia

    Guido Governatori

  4. Department of Philosophy, Tsinghua University, Beijing, China

    Fenrong Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, X., Wang, S., Liu, C., Zhang, M., Liu, X., Xie, H. (2021). Thinking in Patch: Towards Generalizable Forgery Detection with Patch Transformation. In: Pham, D.N., Theeramunkong, T., Governatori, G., Liu, F. (eds) PRICAI 2021: Trends in Artificial Intelligence. PRICAI 2021. Lecture Notes in Computer Science(), vol 13033. Springer, Cham. https://doi.org/10.1007/978-3-030-89370-5_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-89370-5_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-89369-9

  • Online ISBN: 978-3-030-89370-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation