Linghui Luo
ABSTRACT
Integrating static analyses into continuous integration (CI) or continuous delivery (CD) has become the best practice for assuring code quality and security. Static Application Security Testing (SAST) tools fit well into CI/CD, because CI/CD allows time for deep static analyses on large code bases and prevents vulnerabilities in the early stages of the development lifecycle. In CI/CD, the SAST tools usually run in the cloud and provide findings via a web interface. Recent studies show that developers prefer seeing the findings of these tools directly in their IDEs. Most tools with IDE integration run lightweight static analyses and can give feedback at coding time, but SAST tools used in CI/CD take longer to run and usually are not able to do so. Can developers interact directly with a cloud-based SAST tool that is typically used in CI/CD through their IDE? We investigated if such a mechanism can integrate cloud-based SAST tools better into a developers’ workflow than web-based solutions. We interviewed developers to understand their expectations from an IDE solution. Guided by these interviews, we implemented an IDE prototype for an existing cloud-based SAST tool. With a usability test using this prototype, we found that the IDE solution promoted more frequent tool interactions. In particular, developers performed code scans three times more often. This indicates better integration of the cloud-based SAST tool into developers’ workflow. Furthermore, while our study did not show statistically significant improvement on developers’ code-fixing performance, it did show a promising reduction in time for fixing vulnerable code.
- Atlassian. 2021. Gitflow Workflow. https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflowGoogle Scholar
- AWS. 2019. CodeGuru Reviewer. https://aws.amazon.com/codeguruGoogle Scholar
- Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software. In IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering, SANER 2016, Suita, Osaka, Japan, March 14-18, 2016 - Volume 1. IEEE Computer Society, 470–481. https://doi.org/10.1109/SANER.2016.105 Google ScholarCross Ref
- Claude Bolduc. 2016. Lessons learned: Using a static analysis tool within a continuous integration system. In 2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). 37–40. https://doi.org/10.1109/ISSREW.2016.48 Google ScholarCross Ref
- John Brooke. 1996. SUS: A ’Quick and Dirty’ Usability Scale. Usability Evaluation In Industry, 189 (1996), https://doi.org/10.1201/9781498710411-35 Google ScholarCross Ref
- Gary Charness, Uri Gneezy, and Michael A Kuhn. 2012. Experimental methods: Between-subject and within-subject design. Journal of Economic Behavior & Organization, 81, 1 (2012), 1–8. https://doi.org/10.1016/j.jebo.2011.08.009 Google ScholarCross Ref
- Checkmarx. 2021. Checkmarx. https://www.checkmarx.comGoogle Scholar
- Maria Christakis and Christian Bird. 2016. What Developers Want and Need from Program Analysis: An Empirical Study. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE 2016). Association for Computing Machinery, New York, NY, USA. 332–343. isbn:9781450338455 https://doi.org/10.1145/2970276.2970347 Google ScholarDigital Library
- Dino Distefano, Manuel Fähndrich, Francesco Logozzo, and Peter W. O’Hearn. 2019. Scaling Static Analyses at Facebook. Commun. ACM, 62, 8 (2019), July, 62–70. issn:0001-0782 https://doi.org/10.1145/3338112 Google ScholarDigital Library
- Lisa Nguyen Quang Do, James Wright, and Karim Ali. 2020. Why do software developers use static analysis tools? a user-centered study of developer needs and motivations. IEEE Transactions on Software Engineering, https://doi.org/10.1109/TSE.2020.3004525 Google ScholarCross Ref
- Jennifer Fereday and Eimear Muir-Cochrane. 2006. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. International Journal of Qualitative Methods, 5, 1 (2006), 80–92. https://doi.org/10.1177/160940690600500107 arxiv:https://doi.org/10.1177/160940690600500107. Google ScholarCross Ref
- GitHub. 2021. LGTM. https://lgtm.comGoogle Scholar
- Barney G Glaser, Anselm L Strauss, and Elizabeth Strutzel. 1968. The discovery of grounded theory; strategies for qualitative research. Nursing research, 17, 4 (1968), 364. https://doi.org/10.1177/003803856900300233 Google ScholarCross Ref
- Greg Guest, Arwen Bunce, and Laura Johnson. 2006. How many interviews are enough? An experiment with data saturation and variability. Field methods, 18, 1 (2006), 59–82. https://doi.org/10.1177/1525822X05279903 Google ScholarCross Ref
- Nasif Imtiaz, Brendan Murphy, and Laurie Williams. 2019. How do developers act on static analysis alerts? an empirical study of coverity usage. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). 323–333. https://doi.org/10.1109/ISSRE.2019.00040 Google ScholarCross Ref
- Brittany Johnson, Yoonki Song, Emerson R. Murphy-Hill, and Robert W. Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs? In 35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013, David Notkin, Betty H. C. Cheng, and Klaus Pohl (Eds.). IEEE Computer Society, 672–681. https://doi.org/10.1109/ICSE.2013.6606613 Google ScholarCross Ref
- Rahul Kumar, Chetan Bansal, and Jakob Lichtenberg. 2016. Static Analysis Using the Cloud. Electronic Proceedings in Theoretical Computer Science, 228 (2016), Oct, 2–15. issn:2075-2180 https://doi.org/10.4204/eptcs.228.2 Google ScholarCross Ref
- Linghui Luo, Julian Dolby, and Eric Bodden. 2019. MagpieBridge: A General Approach to Integrating Static Analyses into IDEs and Editors (Tool Insights Paper). In 33rd European Conference on Object-Oriented Programming, ECOOP 2019, July 15-19, 2019, London, United Kingdom, Alastair F. Donaldson (Ed.) (LIPIcs, Vol. 134). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 21:1–21:25. https://doi.org/10.4230/LIPIcs.ECOOP.2019.21 Google ScholarCross Ref
- Linghui Luo, Martin Schäf, Daniel Sanchez, and Eric Bodden. 2021. List of codes and their definitions. https://github.com/linghuiluo/FSE21Study/blob/main/ListOfCodes.pdfGoogle Scholar
- Linghui Luo, Martin Schäf, Daniel Sanchez, and Eric Bodden. 2021. List of questions asked in user interviews. https://github.com/linghuiluo/FSE21Study/blob/main/ListOfInterviewQuestions.pdfGoogle Scholar
- Linghui Luo, Martin Schäf, Daniel Sanchez, and Eric Bodden. 2021. Questions in the survey. https://github.com/linghuiluo/FSE21Study/blob/main/SurveyQuestions.pdfGoogle Scholar
- Linghui Luo, Martin Schäf, Daniel Sanchez, and Eric Bodden. 2021. Test applications and issue list. https://github.com/linghuiluo/FSE21Study/tree/main/tasksGoogle Scholar
- Microsoft. 2021. Language Server Protocol. https://microsoft.github.io/language-server-protocolGoogle Scholar
- Graeme D Ruxton and Markus Neuhäuser. 2010. When should we use one-tailed hypothesis testing? Methods in Ecology and Evolution, 1, 2 (2010), 114–117. https://doi.org/10.1111/j.2041-210x.2010.00014.x Google ScholarCross Ref
- Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, and Ciera Jaspan. 2018. Lessons from Building Static Analysis Tools at Google. Commun. ACM, 61, 4 (2018), March, 58–66. issn:0001-0782 https://doi.org/10.1145/3188720 Google ScholarDigital Library
- Douglas Schuler and Aki Namioka. 1993. Participatory design: Principles and practices. CRC Press. https://dl.acm.org/doi/book/10.5555/563076Google ScholarDigital Library
- Amazon Web Services. 2021. AWS SDK for Java. https://aws.amazon.com/sdk-for-javaGoogle Scholar
- Amazon Web Services. 2021. Public API of CodeGuru Reviewer. https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/codegurureviewer/package-summary.htmlGoogle Scholar
- S. S. SHAPIRO and M. B. WILK. 1965. An analysis of variance test for normality (complete samples). Biometrika, 52, 3-4 (1965), dec, 591–611. https://doi.org/10.1093/biomet/52.3-4.591 Google ScholarCross Ref
- Daniela Steidl and Sebastian Eder. 2014. Prioritizing maintainability defects based on refactoring recommendations. In 22nd International Conference on Program Comprehension, ICPC 2014, Hyderabad, India, June 2-3, 2014, Chanchal K. Roy, Andrew Begel, and Leon Moonen (Eds.). ACM, 168–176. https://doi.org/10.1145/2597008.2597805 Google ScholarDigital Library
- Synopsys. 2021. Coverity Scan. https://scan.coverity.comGoogle Scholar
- John C. Tang, Jed R. Brubaker, and Catherine C. Marshall. 2013. What Do You See in the Cloud? Understanding the Cloud-Based User Experience through Practices. In Human-Computer Interaction - INTERACT 2013 - 14th IFIP TC 13 International Conference, Cape Town, South Africa, September 2-6, 2013, Proceedings, Part II, Paula Kotzé, Gary Marsden, Gitte Lindgaard, Janet Wesson, and Marco Winckler (Eds.) (Lecture Notes in Computer Science, Vol. 8118). Springer, 678–695. https://doi.org/10.1007/978-3-642-40480-1_47 Google ScholarCross Ref
- Ilkka Uusitalo, Kaarina Karppinen, Arto Juhola, and Reijo Savola. 2010. Trust and cloud services-an interview study. In 2010 IEEE Second International Conference on Cloud Computing Technology and Science. 712–720. https://doi.org/10.1109/CloudCom.2010.41 Google ScholarDigital Library
- Kaisa Väänänen-Vainio-Mattila, Jarmo Palviainen, Santtu Pakarinen, Else Lagerstam, and Eeva Kangas. 2011. User perceptions of Wow experiences and design implications for Cloud services. In Designing Pleasurable Products and Interfaces, DPPI ’11, Milano, Italy, June 22-25, 2011, Alessandro Deserti, Francesco Zurlo, and Francesca Rizzo (Eds.). ACM, 63:1–63:8. https://doi.org/10.1145/2347504.2347573 Google ScholarDigital Library
- Carmine Vassallo, Fabio Palomba, Alberto Bacchelli, and Harald C Gall. 2018. Continuous code quality: are we (really) doing that? In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 790–795. https://doi.org/10.1145/3238147.3240729 Google ScholarDigital Library
- C. Vassallo, S. Panichella, F. Palomba, S. Proksch, A. Zaidman, and H. C. Gall. 2018. Context is king: The developer perspective on the usage of static analysis tools. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). 38–49. https://doi.org/10.1109/SANER.2018.8330195 Google ScholarCross Ref
- Veracode. 2021. Veracode. https://www.veracode.com/products/binary-static-analysis-sastGoogle Scholar
- Veracode. 2021. Veracode Static For IDE. https://help.veracode.com/r/api_eclipseGoogle Scholar
- Lizhe Wang, Jie Tao, Marcel Kunze, Alvaro Canales Castellanos, David Kramer, and Wolfgang Karl. 2008. Scientific Cloud Computing: Early Definition and Experience. In 10th IEEE International Conference on High Performance Computing and Communications, HPCC 2008, 25-27 Sept. 2008, Dalian, China. IEEE Computer Society, 825–830. https://doi.org/10.1109/HPCC.2008.38 Google ScholarCross Ref
- R. F. Woolson. 2008. Wilcoxon Signed-Rank Test. American Cancer Society, 1–3. isbn:9780471462422 https://doi.org/10.1002/9780471462422.eoct979 arxiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780471462422.eoct979. Google ScholarCross Ref
- Fiorella Zampetti, Simone Scalabrino, Rocco Oliveto, Gerardo Canfora, and Massimiliano Di Penta. 2017. How open source projects use static code analysis tools in continuous integration pipelines. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). 334–344. https://doi.org/10.1109/MSR.2017.2 Google ScholarDigital Library
Index Terms
- IDE support for cloud-based static analyses
Recommendations
Are REST APIs for Cloud Computing Well-Designed? An Exploratory Study
Service-Oriented ComputingAbstractCloud computing is currently the most popular model to offer and access computational resources and services. Many cloud providers use the REST architectural style (Representational State Transfer) for offering such computational resources. ...
Static determination of allocation rates to support real-time garbage collection
LCTES '05: Proceedings of the 2005 ACM SIGPLAN/SIGBED conference on Languages, compilers, and tools for embedded systemsWhile it is generally accepted that garbage-collected languages offer advantages over languages in which objects must be explicitly deallocated, real-time developers are leery of the adverse effects a garbage collector might have on real-time ...
Comments