Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enriching Automatic Test Case Generation by Extracting Relevant Test Inputs from Bug Reports (2312.14898v2)

Published 22 Dec 2023 in cs.SE

Abstract: The quality of software is closely tied to the effectiveness of the tests it undergoes. Manual test writing, though crucial for bug detection, is time-consuming, which has driven significant research into automated test case generation. However, current methods often struggle to generate relevant inputs, limiting the effectiveness of the tests produced. To address this, we introduce BRMiner, a novel approach that leverages LLMs in combination with traditional techniques to extract relevant inputs from bug reports, thereby enhancing automated test generation tools. In this study, we evaluate BRMiner using the Defects4J benchmark and test generation tools such as EvoSuite and Randoop. Our results demonstrate that BRMiner achieves a Relevant Input Rate (RIR) of 60.03% and a Relevant Input Extraction Accuracy Rate (RIEAR) of 31.71%, significantly outperforming methods that rely on LLMs alone. The integration of BRMiner's input enhances EvoSuite ability to generate more effective test, leading to increased code coverage, with gains observed in branch, instruction, method, and line coverage across multiple projects. Furthermore, BRMiner facilitated the detection of 58 unique bugs, including those that were missed by traditional baseline approaches. Overall, BRMiner's combination of LLM filtering with traditional input extraction techniques significantly improves the relevance and effectiveness of automated test generation, advancing the detection of bugs and enhancing code coverage, thereby contributing to higher-quality software development.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. An industrial evaluation of unit test generation: Finding real faults in a financial application. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pages 263–272. IEEE, 2017.
  2. Parameter tuning or default values? an empirical investigation in search-based software engineering. Empirical Software Engineering, 18:594–623, 2013.
  3. A framework for automated testing of javascript web applications. In Proceedings of the 33rd International Conference on Software Engineering, pages 571–580, 2011.
  4. A survey of symbolic execution techniques. ACM Computing Surveys (CSUR), 51(3):1–39, 2018.
  5. Extracting structural information from bug reports. In Proceedings of the 2008 international working conference on Mining software repositories, pages 27–30, 2008.
  6. Automatically generating realistic test input from web services. In Proceedings of 2011 IEEE 6th International Symposium on Service Oriented System (SOSE), pages 13–24. IEEE, 2011.
  7. Exe: Automatically generating inputs of death. ACM Transactions on Information and System Security (TISSEC), 12(2):1–38, 2008.
  8. Symbolic execution for software testing: three decades later. Communications of the ACM, 56(2):82–90, 2013.
  9. Heike Wehrheim Cedric Richter. Tssb-3m: Mining single statement bugs at massive scale. In MSR, 2022.
  10. Improving web application testing with user session data. In 25th International Conference on Software Engineering, 2003. Proceedings., pages 49–59. IEEE, 2003.
  11. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 416–419, 2011.
  12. Evosuite at the sbst 2013 tool competition. pages 406–409, 03 2013.
  13. Does automated unit test generation really help software testers? a controlled empirical study. ACM Transactions on Software Engineering and Methodology (TOSEM), 24(4):1–49, 2015.
  14. Improving search-based test suite generation with dynamic symbolic execution. In 2013 ieee 24th international symposium on software reliability engineering (issre), pages 360–369. IEEE, 2013.
  15. Extending a search-based test generator with adaptive dynamic symbolic execution. In Proceedings of the 2014 international symposium on software testing and analysis, pages 421–424, 2014.
  16. Dart: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, pages 213–223, 2005.
  17. Perflearner: Learning from bug reports to understand and generate performance test frames. In Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, pages 17–28, 2018.
  18. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering, 36(2):226–247, 2010.
  19. Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 international symposium on software testing and analysis, pages 437–440, 2014.
  20. James C King. Symbolic execution and program testing. Communications of the ACM, 19(7):385–394, 1976.
  21. Adoption of software testing in open source projects–a preliminary study on 50,000 projects. In 2013 17th european conference on software maintenance and reengineering, pages 353–356. IEEE, 2013.
  22. An empirical study of adoption of software testing in open source projects. In 2013 13th International Conference on Quality Software, pages 103–112. IEEE, 2013.
  23. Understanding the test automation culture of app developers. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pages 1–10. IEEE, 2015.
  24. JaCoCo Java Code Coverage Library. https://www.jacoco.org/jacoco/.
  25. Automatic text input generation for mobile testing. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pages 643–653. IEEE, 2017.
  26. Directed test generation using symbolic grammars. In Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering, pages 134–143, 2007.
  27. Link: exploiting the web of data to generate test inputs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 373–384, 2014.
  28. Search-based test input generation for string data types using the results of web queries. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pages 141–150. IEEE, 2012.
  29. Leveraging existing tests in automated test generation for web applications. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering, pages 67–78, 2014.
  30. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE’07), pages 75–84. IEEE, 2007.
  31. Reformulating branch coverage as a many-objective optimization problem. In 2015 IEEE 8th international conference on software testing, verification and validation (ICST), pages 1–10. IEEE, 2015.
  32. Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Transactions on Software Engineering, 44(2):122–158, 2017.
  33. Defect prediction guided search-based software testing. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pages 448–460, 2020.
  34. Fully automatic and precise detection of thread safety violations. In Proceedings of the 33rd ACM SIGPLAN conference on Programming Language Design and Implementation, pages 521–530, 2012.
  35. Configuring test generators using bug reports: a case study of gcc compiler and csmith. In Proceedings of the 36th Annual ACM Symposium on Applied Computing, pages 1750–1758, 2021.
  36. Cute and jcute: Concolic unit testing and explicit path model-checking tools: (tool paper). In Computer Aided Verification: 18th International Conference, CAV 2006, Seattle, WA, USA, August 17-20, 2006. Proceedings 18, pages 419–423. Springer, 2006.
  37. Cute: A concolic unit testing engine for c. ACM SIGSOFT Software Engineering Notes, 30(5):263–272, 2005.
  38. Automated discovery of valid test strings from the web using dynamic regular expressions collation and natural language processing. In 2012 12th International Conference on Quality Software, pages 79–88. IEEE, 2012.
  39. Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 201–211. IEEE, 2015.
  40. Generation of string test input from web using regular expression. International Journal of Computer Applications, 975:8887, 2014.
  41. Saying ’hi!’ is not enough: mining inputs for effective test generation. In Proceedings of the 32nd International Conference on Automated Software Engineering, pages 44–49. IEEE Computer Society, 2017.
  42. Mutation-inspired symbolic execution for software testing. IET Software, 16(5):478–492, 2022.
  43. Symstra: A framework for generating object-oriented unit tests using symbolic execution. In Tools and Algorithms for the Construction and Analysis of Systems: 11th International Conference, TACAS 2005, Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2005, Edinburgh, UK, April 4-8, 2005. Proceedings 11, pages 365–381. Springer, 2005.
  44. Hao Zhong. Enriching compiler testing with real program from bug report. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–12, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets