How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts (2404.17739v2)

Abstract: Since the emergence of GPT-3, LLMs have caught the eyes of researchers, practitioners, and educators in the field of software engineering. However, there has been relatively little investigation regarding the performance of LLMs in assisting with requirements analysis and UML modeling. This paper explores how LLMs can assist novice analysts in creating three types of typical UML models: use case models, class diagrams, and sequence diagrams. For this purpose, we designed the modeling tasks of these three UML models for 45 undergraduate students who participated in a requirements modeling course, with the help of LLMs. By analyzing their project reports, we found that LLMs can assist undergraduate students as novice analysts in UML modeling tasks, but LLMs also have shortcomings and limitations that should be considered when using them.

• The paper demonstrates that LLMs support UML model drafting, achieving 88.89% correctness in identifying use cases and 82.22% in sequencing messages.
• The study found that while LLMs perform moderately in class diagram creation (66.67% for classes, 75.56% for operations), they struggle with identifying relationships (24.44% correctness).
• Hybrid-created diagrams, which combine AI generation with human refinement, outperformed other formats, underscoring the importance of human oversight.

How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts

Introduction

The study "How LLMs Aid in UML Modeling: An Exploratory Study with Novice Analysts" explores the capacity of LLMs to assist undergraduate students in creating UML models, specifically use case diagrams, class diagrams, and sequence diagrams. This investigation comes in the context of a requirements modeling course involving 45 participants, aiming to understand the practical impact of LLMs in software engineering tasks.

Experimentation and Design

The experimental design involved a structured task where students used LLMs, predominantly ChatGPT, to aid the creation of UML diagrams for a given case study. Each participant submitted a project report comprising the UML models generated and the transcript of interactions with LLMs.

Figure 1: The process of the experiment.

Results of UML Model Creation

Use Case Modeling

In evaluating the use case models generated with LLM assistance, several insights were evident:

• LLMs excelled in identifying use cases correctly with a high success rate, as seen in 88.89% correctness.
• However, the identification of actors and their relationships was notably less effective, achieving only 31.11% and 17.78% correctness, respectively.

Class Diagram Modeling

For class diagram creation, LLMs demonstrated good performance in identifying classes and operations, with correctness rates of 66.67% and 75.56%, respectively.

Figure 2: Distribution of the participants with/without experience of using LLMs.

• The recognition of relationships among classes presented challenges, with a correctness rate of merely 24.44%.

Sequence Diagram Modeling

Sequence diagrams benefitted from LLM assistance in recognizing objects and sequencing messages, where the correctness of object identification reached 73.33%.

• Correct sequence ordering achieved 82.22% correctness, indicating a capacity for LLMs to comprehend and arrange chronological activities effectively.

Output Formats and Analysis

The research further delved into the output formats utilized in UML creation:

• Hybrid-created diagrams performed best, with an average score of 8.20, showcasing the significant role of human intervention and optimization.
• PlantUML-based diagrams had a moderate performance (average score 6.94), benefitting from auto-generated code but still requiring manual correction.
• Simple wireframe outputs were the least effective, with an average score of 5.5, often lacking the necessary detail and accuracy.

  Figure 3: Distribution of the LLMs used in UML modeling tasks.

Discussion

This study highlights that while LLMs are capable of aiding in software modeling, substantial limitations persist. LLMs often struggle with identifying complex relationships, underscoring a need for further enhancement in understanding relational constructs. These findings are pivotal for educators and industry professionals, suggesting that while LLMs serve as useful tools, reliance on them for complete accuracy without human intervention is premature.

Implications for Software Engineering

The implications for software engineering education are profound. LLMs can be integrated as supplementary tools in teaching UML modeling, leveraging their capacity to generate initial drafts of models but requiring critical human oversight. Educators and professionals must focus on training students to effectively collaborate with LLMs, enhancing their understanding while avoiding blind reliance on AI-generated outputs.

Figure 4: Distribution of the languages used in the human-LLM interaction.

Conclusion

The exploratory study demonstrates that LLMs hold potential in assisting novice engineers with UML modeling tasks but still have significant shortcomings in relational analysis and diagram precision. As AI continues to evolve, ongoing research and refinements are essential to transform LLMs into reliable partners in software engineering practices.