Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A study on the impact of pre-trained model on Just-In-Time defect prediction (2309.02317v2)

Published 5 Sep 2023 in cs.SE and cs.LG

Abstract: Previous researchers conducting Just-In-Time (JIT) defect prediction tasks have primarily focused on the performance of individual pre-trained models, without exploring the relationship between different pre-trained models as backbones. In this study, we build six models: RoBERTaJIT, CodeBERTJIT, BARTJIT, PLBARTJIT, GPT2JIT, and CodeGPTJIT, each with a distinct pre-trained model as its backbone. We systematically explore the differences and connections between these models. Specifically, we investigate the performance of the models when using Commit code and Commit message as inputs, as well as the relationship between training efficiency and model distribution among these six models. Additionally, we conduct an ablation experiment to explore the sensitivity of each model to inputs. Furthermore, we investigate how the models perform in zero-shot and few-shot scenarios. Our findings indicate that each model based on different backbones shows improvements, and when the backbone's pre-training model is similar, the training resources that need to be consumed are much more closer. We also observe that Commit code plays a significant role in defect detection, and different pre-trained models demonstrate better defect detection ability with a balanced dataset under few-shot scenarios. These results provide new insights for optimizing JIT defect prediction tasks using pre-trained models and highlight the factors that require more attention when constructing such models. Additionally, CodeGPTJIT and GPT2JIT achieved better performance than DeepJIT and CC2Vec on the two datasets respectively under 2000 training samples. These findings emphasize the effectiveness of transformer-based pre-trained models in JIT defect prediction tasks, especially in scenarios with limited training data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. S. Kim, E. J. Whitehead, and Z. Yi, “Classifying software changes: Clean or buggy?” IEEE Transactions on Software Engineering, vol. 34, no. 2, pp. 181–196, 2008.
  2. T. Hoang, H. Khanh Dam, Y. Kamei, D. Lo, and N. Ubayashi, “Deepjit: An end-to-end deep learning framework for just-in-time defect prediction,” in 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), 2019, pp. 34–45.
  3. T. Hoang, H. J. Kang, D. Lo, and J. Lawall, “Cc2vec: Distributed representations of code changes,” in 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), 2020, pp. 518–529.
  4. J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “Bert: pre-trained of deep bidirectional transformers for language understanding,” 2018.
  5. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, and D. Jiang, “Codebert: A pre-trained model for programming and natural languages,” 2020.
  6. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.
  7. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” 2019.
  8. W. U. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” 2021.
  9. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res., vol. 21, pp. 140:1–140:67, 2020.
  10. M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line-level vulnerability prediction,” in 19th IEEE/ACM International Conference on Mining Software Repositories, MSR 2022, Pittsburgh, PA, USA, May 23-24, 2022. ACM, 2022, pp. 608–620.
  11. X. Zhou, D. Han, and D. Lo, “Assessing generalizability of codebert,” in 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2021, pp. 425–436.
  12. N. Pappas and A. Popescu-Belis, “Multilingual hierarchical attention networks for document classification,” 2017.
  13. T. Mikolov, I. Sutskever, C. Kai, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in arXiv, 2013.
  14. T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” Computer Science, 2013.
  15. J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in Conference on Empirical Methods in Natural Language Processing, 2014.
  16. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in arXiv, 2017.
  17. M. Peters, M. Neumann, M. Iyyer, M. Gardner, and L. Zettlemoyer, “Deep contextualized word representations,” 2018.
  18. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019.
  19. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever et al., “Improving language understanding by generative pre-trained,” 2018.
  20. S. McIntosh and Y. Kamei, “Are fix-inducing changes a moving target?: A longitudinal case study of just-in-time defect prediction,” in Proceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY, USA: ACM, 2018, pp. 560-560.
  21. A. Mockus and D. M. Weiss, “Predicting risk of software changes,” Bell Labs Technical Journal, vol. 5, no. 2, pp. 169–180, 2000.
  22. Mockus and Votta, “Identifying reasons for software changes using historic databases,” in Proc of International Conference on Software Maintenance, 2000, pp. 120–130.
  23. N. F. Schneidewind and H. M. Hoffmann, “An experiment in software error data collection and analysis,” IEEE Trans Software Engineering, vol. 5, no. 3, pp. 276–286, 1979.
  24. M. Tan, L. Tan, S. Dara, and C. Mayeux, “Online defect prediction for imbalanced data,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2, 2015, pp. 99–108.
  25. Y. Kamei, T. Fukushima, S. Mcintosh, K. Yamashita, N. Ubayashi, and A. E. Hassan, “Studying just-in-time defect prediction using cross-project models,” Empirical Software Engineering, vol. 21, no. 5, pp. 2072–2106, 2016.
  26. S. McIntosh and Y. Kamei, “[journal first] are fix-inducing changes a moving target?: A longitudinal case study of just-in-time defect prediction,” in 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), 2018, pp. 560–560.
  27. T. M. Khoshgoftaar and E. B. Allen, “Ordering fault-prone software modules,” vol. 11, no. 1, pp. 19–37, 2003.
  28. T. Gyimothy, R. Ferenc, and I. Siket, “Empirical validation of object-oriented metrics on open source software for fault prediction,” IEEE Transactions on Software Engineering, vol. 31, no. 10, pp. 897–910, 2005.
  29. J. Sliwerski, T. Zimmermann, and A. Zeller, “When do changes induce fixes?” in Proceedings of the 2005 International Workshop on Mining Software Repositories, MSR 2005, Saint Louis, Missouri, USA, May 17, 2005, 2005.
  30. S. Kim, T. Zimmermann, P. Kai, and E. Jr, “Automatic identification of bug-introducing changes,” in Proc International Conference on Automated Software Engineering, 2006.
  31. D. Costa, S. Mcintosh, W. Shang, U. Kulesza, R. Coelho, and A. E. Hassan, “A framework for evaluating the results of the szz approach for identifying bug-introducing changes,” IEEE TRANSACTIONS ON SOFTWARE ENGINEERING SE, vol. 43, no. 7, pp. 641–657, 2017.
  32. Hinton and E. Geoffrey, “How neural networks learn from experience.” Scientific American, vol. 267, no. 3, pp. 144–51, 1992.
  33. Y. Bengio, 2009.
  34. G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Computation, vol. 18, no. 7, pp. 1527–1554, 2006.
  35. A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, no. 2, 2012.
  36. S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in JMLR.org, 2015.
  37. Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, p. 436, 2015.
  38. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  39. X. Yang, D. Lo, X. Xia, Y. Zhang, and J. Sun, “Deep learning for just-in-time defect prediction,” in 2015 IEEE International Conference on Software Quality, Reliability and Security, 2015, pp. 17–26.
  40. C. Pornprasit and C. K. Tantithamthavorn, “Jitline: A simpler, better, faster, finer-grained just-in-time defect prediction,” in 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), 2021, pp. 369–379.
  41. M. Yan, X. Xia, Y. Fan, A. E. Hassan, D. Lo, and S. Li, “Just-in-time defect identification and localization: A two-phase framework,” IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 82–101, 2022.
  42. W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent neural network regularization,” Eprint Arxiv, 2014.
  43. Z. Zeng, Y. Zhang, H. Zhang, and L. Zhang, “Deep just-in-time defect prediction: how far are we?” in ISSTA ’21: 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021.
Citations (2)

Summary

We haven't generated a summary for this paper yet.