skip to main content
10.1145/3359789.3359812acmotherconferencesArticle/Chapter ViewAbstractPublication PagesacsacConference Proceedingsconference-collections
research-article

How to kill symbolic deobfuscation for free (or: unleashing the potential of path-oriented protections)

Published:09 December 2019Publication History

ABSTRACT

Code obfuscation is a major tool for protecting software intellectual property from attacks such as reverse engineering or code tampering. Yet, recently proposed (automated) attacks based on Dynamic Symbolic Execution (DSE) shows very promising results, hence threatening software integrity. Current defenses are not fully satisfactory, being either not efficient against symbolic reasoning, or affecting runtime performance too much, or being too easy to spot. We present and study a new class of anti-DSE protections coined as path-oriented protections targeting the weakest spot of DSE, namely path exploration. We propose a lightweight, efficient, resistant and analytically proved class of obfuscation algorithms designed to hinder DSE-based attacks. Extensive evaluation demonstrates that these approaches critically counter symbolic deobfuscation while yielding only a very slight overhead.

References

  1. Tigress challenge. http://tigress.cs.arizona.edu/challenges.html.Google ScholarGoogle Scholar
  2. S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, and P. McMinn. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 2013.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Thanassis Avgerinos, Alexandre Rebert, Sang Kil Cha, and David Brumley. Enhancing symbolic execution with veritesting. Commun. ACM, 59(6), 2016.Google ScholarGoogle Scholar
  4. Gogul Balakrishnan and Thomas W. Reps. WYSINWYX: what you see is not what you execute. ACM Trans. Program. Lang. Syst., 32, 2010.Google ScholarGoogle Scholar
  5. Sebastian Banescu, Christian S. Collberg, Vijay Ganesh, Zack Newsham, and Alexander Pretschner. Code obfuscation against symbolic execution attacks. In Annual Conference on Computer Security Applications, ACSAC 2016, 2016.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Sebastian Banescu, Christian S. Collberg, and Alexander Pretschner. Predicting the resilience of obfuscated code against symbolic execution attacks via machine learning. In USENIX Security Symposium, 2017.Google ScholarGoogle Scholar
  7. Boaz Barak, Oded Goldreich, Russell Impagliazzo, Steven Rudich, Amit Sahai, Salil P. Vadhan, and Ke Yang. On the (im)possibility of obfuscating programs. In Advances in Cryptology - CRYPTO, 2001.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Sébastien Bardin, Robin David, and Jean-Yves Marion. Backward-bounded DSE: targeting infeasibility questions on obfuscated codes. In 2017 IEEE Symposium on Security and Privacy, SP, 2017.Google ScholarGoogle ScholarCross RefCross Ref
  9. Clark Barrett and Cesare Tinelli. Satisfiability Modulo Theories. Springer International Publishing, 2018.Google ScholarGoogle ScholarCross RefCross Ref
  10. Armin Biere. Bounded Model Checking. In Handbook of Satisfiability. 2009.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Fabrizio Biondi, Sébastien Josse, Axel Legay, and Thomas Sirvent. Effectiveness of synthesis in concolic deobfuscation. Computers & Security, 70, 2017.Google ScholarGoogle Scholar
  12. Guillaume Bonfante, José M. Fernandez, Jean-Yves Marion, Benjamin Rouxel, Fabrice Sabatier, and Aurélien Thierry. Codisasm: Medium scale concatic disassembly of self-modifying binaries with overlapping instructions. In Conference on Computer and Communications Security, 2015.Google ScholarGoogle Scholar
  13. David Brumley, Cody Hartwig, Zhenkai Liang, James Newsome, Dawn Xiaodong Song, and Heng Yin. Automatically identifying trigger-based behavior in malware. In Wenke Lee, Cliff Wang, and David Dagon, editors, Botnet Detection: Countering the Largest Security Threat, volume 36 of Advances in Information Security, pages 65--88. Springer, 2008.Google ScholarGoogle Scholar
  14. Robert Brummayer and Armin Biere. Boolector: An efficient SMT solver for bit-vectors and arrays. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS, 2009.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Roberto Bruni, Roberto Giacobazzi, and Roberta Gori. Code obfuscation against abstract model checking attacks. In Verification, Model Checking, and Abstract Interpretation - 19th International Conference, VMCAI, 2018.Google ScholarGoogle Scholar
  16. Cristian Cadar. Targeted program transformations for symbolic execution. In Meeting on Foundations of Software Engineering, ESEC/FSE, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI, 2008.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Cristian Cadar and Koushik Sen. Symbolic execution for software testing: three decades later. Commun. ACM, 56(2), 2013.Google ScholarGoogle Scholar
  19. Mariano Ceccato, Paolo Tonella, Cataldo Basile, Paolo Falcarin, Marco Torchiano, Bart Coppens, and Bjorn De Sutter. Understanding the behaviour of hackers while performing attack tasks in a professional setting and in a public challenge. Empirical Software Engineering, 24(1):240--286, Feb 2019.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Sang Kil Cha, Thanassis Avgerinos, Alexandre Rebert, and David Brumley. Unleashing mayhem on binary code. In Symposium on Security and Privacy, SP, 2012.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Christian Collberg and Jasvir Nagra. Surreptitious Software: Obfuscation, Watermarking, and Tamperproofing for Software Protection. Addison-Wesley Professional, 1st edition, 2009.Google ScholarGoogle Scholar
  22. Christian Collberg, Clark Thomborson, and Douglas Low. A taxonomy of obfuscating transformations, 1997.Google ScholarGoogle Scholar
  23. Christian S. Collberg, Sam Martin, Jonathan Myers, and Jasvir Nagra. Distributed application tamper detection via continuous software updates. In Annual Computer Security Applications Conference, ACSAC, 2012.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Kevin Coogan, Gen Lu, and Saumya K. Debray. Deobfuscation of virtualization-obfuscated software: a semantics-based approach. In Conference on Computer and Communications Security, CCS, 2011.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Robin David, Sébastien Bardin, Josselin Feist, Laurent Mounier, Marie-Laure Potet, Thanh Dinh Ta, and Jean-Yves Marion. Specification of concretization and symbolization policies in symbolic execution. In International Symposium on Software Testing and Analysis, ISSTA 2016, 2016.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Robin David, Sébastien Bardin, Thanh Dinh Ta, Laurent Mounier, Josselin Feist, Marie-Laure Potet, and Jean-Yves Marion. BINSEC/SE: A dynamic symbolic execution toolkit for binary-level analysis. In IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, SANER, 2016.Google ScholarGoogle ScholarCross RefCross Ref
  27. Leonardo Mendonça de Moura and Nikolaj Bjørner. Z3: an efficient SMT solver. In Tools and Algorithms for the Construction and Analysis of Systems, TACAS, 2008.Google ScholarGoogle Scholar
  28. Saumya K. Debray and Jay Patel. Reverse engineering self-modifying code: Unpacker extraction. In Working Conference on Reverse Engineering, WCRE, 2010.Google ScholarGoogle Scholar
  29. Ninon Eyrolles, Louis Goubin, and Marion Videau. Defeating mba-based obfuscation. In Proceedings of the 2016 ACM Workshop on Software PROtection, SPRO@CCS 2016, 2016.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Patrice Godefroid, Michael Y. Levin, and David A. Molnar. SAGE: whitebox fuzzing for security testing. Commun. ACM, 55(3), 2012.Google ScholarGoogle Scholar
  31. Thomas A. Henzinger, Ranjit Jhala, Rupak Majumdar, and Grégoire Sutre. Lazy abstraction. In The 29th SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), 2002.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Min Gyung Kang, Pongsin Poosankam, and Heng Yin. Renovo: a hidden code extractor for packed executables. In ACM Workshop Recurring Malcode (WORM). ACM, 2007.Google ScholarGoogle Scholar
  33. Johannes Kinder. Towards static analysis of virtualization-obfuscated binaries. In 19th Working Conference on Reverse Engineering, WCRE, 2012.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Dave King, Boniface Hicks, Michael Hicks, and Trent Jaeger. Implicit flows: Can't live with 'em, can't live without 'em. In Information Systems Security, 4th International Conference, ICISS, 2008.Google ScholarGoogle Scholar
  35. Yin Liu and Ana Milanova. Static information flow analysis with handling of implicit flows and a study on effects of implicit flows vs explicit flows. In 14th European Conference on Software Maintenance and Reengineering, CSMR, 2010.Google ScholarGoogle Scholar
  36. Saeed Nejati, Jia Hui Liang, Catherine H. Gebotys, Krzysztof Czarnecki, and Vijay Ganesh. Adaptive restart and cegar-based solver for inverting cryptographic hash functions. In VSTTE, 2017.Google ScholarGoogle ScholarCross RefCross Ref
  37. Jonathan Salwan, Sébastien Bardin, and Marie-Laure Potet. Symbolic deobfuscation: from virtualized code back to the original. In 5th Conference on Detection of Intrusions and malware & Vulnerability Assessment (DIMVA), 2018.Google ScholarGoogle ScholarCross RefCross Ref
  38. Florent Saudel and Jonathan Salwan. Triton : Framework d'exÃl'cution concolique. In SSTIC, 2015.Google ScholarGoogle Scholar
  39. Sebastian Schrittwieser, Stefan Katzenbeisser, Johannes Kinder, Georg Merzdovnik, and Edgar Weippl. Protecting software through obfuscation: Can it keep pace with progress in code analysis? ACM Comput. Surv., 49(1), 2016.Google ScholarGoogle Scholar
  40. Edward J. Schwartz, Thanassis Avgerinos, and David Brumley. All you ever wanted to know about dynamic taint analysis and forward symbolic execution (but might have been afraid to ask). In Symposium on Security and Privacy, S&P, 2010.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Hovav Shacham. The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86). In Conference on Computer and Communications Security, CCS, 2007.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Monirul I. Sharif, Andrea Lanzi, Jonathon T. Giffin, and Wenke Lee. Impeding malware analysis using conditional code obfuscation. In Network and Distributed System Security Symposium, NDSS, 2008.Google ScholarGoogle Scholar
  43. Yan Shoshitaishvili, Ruoyu Wang, Christopher Salls, Nick Stephens, Mario Polino, Andrew Dutcher, John Grosen, Siji Feng, Christophe Hauser, Christopher Krügel, and Giovanni Vigna. SOK: (state of) the art of war: Offensive techniques in binary analysis. In IEEE Symposium on Security and Privacy, SP, 2016.Google ScholarGoogle ScholarCross RefCross Ref
  44. Venkatesh Srinivasan and Thomas W. Reps. An improved algorithm for slicing machine code. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016. ACM.Google ScholarGoogle Scholar
  45. Jon Stephens, Babak Yadegari, Christian S. Collberg, Saumya Debray, and Carlos Scheidegger. Probabilistic obfuscation through covert channels. In European Symposium on Security and Privacy, EuroS&P, 2018.Google ScholarGoogle ScholarCross RefCross Ref
  46. Julien Vanegue and Sean Heelan. SMT solvers in software security. In 6th USENIX Workshop on Offensive Technologies, WOOT'12, 2012.Google ScholarGoogle Scholar
  47. Chenxi Wang, Jonathan Hill, John Knight, and Jack Davidson. Software tamper resistance: Obstructing static analysis of programs. Technical report, Charlottesville, VA, USA, 2000.Google ScholarGoogle Scholar
  48. Chenxi Wang, Jonathan Hill, John C. Knight, and Jack W. Davidson. Protection of software-based survivability mechanisms. In International Conference on Dependable Systems and Networks (DSN), 2001.Google ScholarGoogle ScholarCross RefCross Ref
  49. Zhi Wang, Jiang Ming, Chunfu Jia, and Debin Gao. Linear obfuscation to combat symbolic execution. In European Symposium on Research in Computer Security, ESORICS, 2011.Google ScholarGoogle ScholarCross RefCross Ref
  50. Babak Yadegari and Saumya Debray. Symbolic execution of obfuscated code. In Conference on Computer and Communications Security (CCS), 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Babak Yadegari, Brian Johannesmeyer, Ben Whitely, and Saumya Debray. A generic approach to automatic deobfuscation of executable code. In Symposium on Security and Privacy, SP, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Yongxin Zhou, Alec Main, Yuan Xiang Gu, and Harold Johnson. Information hiding in software with mixed boolean-arithmetic transforms. In Information Security Applications, WISA, 2007.Google ScholarGoogle Scholar

Index Terms

  1. How to kill symbolic deobfuscation for free (or: unleashing the potential of path-oriented protections)

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in
          • Published in

            cover image ACM Other conferences
            ACSAC '19: Proceedings of the 35th Annual Computer Security Applications Conference
            December 2019
            821 pages
            ISBN:9781450376280
            DOI:10.1145/3359789

            Copyright © 2019 ACM

            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 9 December 2019

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article

            Acceptance Rates

            ACSAC '19 Paper Acceptance Rate60of266submissions,23%Overall Acceptance Rate104of497submissions,21%

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader