Which AI Technique Is Better to Classify Requirements? An Experiment with SVM, LSTM, and ChatGPT (2311.11547v2)
Abstract: Recently, LLMs like ChatGPT have demonstrated remarkable proficiency in various Natural Language Processing tasks. Their application in Requirements Engineering, especially in requirements classification, has gained increasing interest. This paper reports an extensive empirical evaluation of two ChatGPT models, specifically gpt-3.5-turbo, and gpt-4 in both zero-shot and few-shot settings for requirements classification. The question arises as to how these models compare to traditional classification methods, specifically Support Vector Machine and Long Short-Term Memory. Based on five different datasets, our results show that there is no single best technique for all types of requirement classes. Interestingly, the few-shot setting has been found to be beneficial primarily in scenarios where zero-shot results are significantly low.
- Non-functional requirements in software engineering, in: A. Borgida, V. Chaudhri, P. Giorgini, E. Yu (Eds.), Conceptual Modeling: Foundations and Applications, volume 5 of Lecture Notes in Computer Science, Springer, 2012.
- Are "non-functional" requirements really non-functional? an investigation of non-functional requirements in practice, in: Proceedings of the IEEE/ACM International Conference on Software Engineering, IEEE, 2016.
- Non-functional requirements as qualities, with a spice of ontology, in: Proceedings of the IEEE International Requirements Engineering Conference, IEEE, 2014.
- Requirements classification with interpretable machine learning and dependency parsing, in: Proceedings of the 2019 IEEE 27th International Requirements Engineering Conference (RE), IEEE, 2019.
- Z. Kurtanović, W. Maalej, Automatically classifying functional and non-functional requirements using supervised machine learning, in: Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference (RE), IEEE, 2017.
- Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc., 2020.
- D. M. Berry, Empirical evaluation of tools for hairy requirements engineering tasks, Empirical Software Engineering 26 (2021) 111.
- Ontology-based classification of non-functional requirements in software specifications: A new corpus and svm-based classifier, in: Proceedings of the 2013 IEEE 37th Annual Computer Software and Applications Conference (COMPSAC), IEEE, 2013.
- aerobert-classifier: Classification of aerospace requirements using bert, Aerospace 10 (2023) 279.
- An ensemble machine learning technique for functional requirement classification, Symmetry 12 (2020) 1601.
- F. Yucalar, Developing an advanced software requirements classification model using bert: An empirical evaluation study on newly generated turkish data, Applied Sciences 13 (2023) 11127.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.