40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study (2206.11102v2)

Abstract: The relevance of code comprehension in a developer's daily work was recognized more than 40 years ago. Consequently, many experiments were conducted to find out how developers could be supported during code comprehension and which code characteristics contribute to better comprehension. Today, such studies are more common than ever. While this is great for advancing the field, the number of publications makes it difficult to keep an overview. Additionally, designing rigorous code comprehension experiments with human participants is a challenging task, and the multitude of design options can make it difficult for researchers, especially newcomers to the field, to select a suitable design. We therefore conducted a systematic mapping study of 95 source code comprehension experiments published between 1979 and 2019. By structuring the design characteristics of code comprehension studies, we provide a basis for subsequent discussion of the huge diversity of design options in the face of a lack of basic research on their consequences and comparability. We describe what topics have been studied, as well as how these studies have been designed, conducted, and reported. Frequently chosen design options and deficiencies are pointed out to support researchers of all levels of domain expertise in designing their own studies.

• The paper systematically maps 40 years of code comprehension experiments, identifying design trends and research gaps.
• It reveals a shift from one-at-a-time factor designs to more balanced factorial approaches, with increasing use of psycho-physiological measures.
• The study recommends standardizing methodologies and diversifying participant samples to bolster reproducibility and external validity.

40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study

Overview and Methodological Insights

The paper presents a systematic mapping paper investigating the design and implementation of code comprehension experiments over four decades. The main objective is to synthesize the diversity of paper designs and discuss the strengths, weaknesses, and gaps in current research practices. The authors selected 95 studies with human participants from 1979 to 2019 to analyze trends in research questions, constructs, tasks, and methodologies.

Figure 1: Schematic representation of the research methodology.

Trends and Themes

The majority of studies (71%) examine how external factors influence code comprehension (X $\Rightarrow$ code comprehension). Surprisingly, none of the studies investigate the impact of code comprehension on other constructs (code comprehension $\Rightarrow$ X), indicating a potential gap in research exploring the practical implications of improved code comprehensibility.

Thematic analysis reveals frequent studies on semantic cues (e.g., identifier naming), developer characteristics, and code structure, with a notable rise in studies utilizing psycho-physiological measures in recent years, reflecting the increased interest in understanding the cognitive aspects of code comprehension.

Figure 2: Evolution of the four most popular paper categories according to their number of occurrences.

Study Design Considerations

Research Methodologies

Experiments dominate the research methods utilized, with a considerable variation in factor designs and allocation strategies. Most studies employ one-at-a-time factor designs paired with within-subject allocation. However, a shift toward factorial designs and balanced group allocations is observed over time, reflecting an evolving understanding of the need for comprehensive paper designs.

Figure 3: Distribution of factor and allocation design.

Participant Selection and Setting

Studies vary significantly in participant demographics, with a predominant use of students. The choice of programming languages reflects industry trends, with Java being the most utilized language. The paper highlights the need for more diverse participant samples, incorporating professionals to improve external validity.

Task Designs and Measures

Tasks such as answering comprehension questions and determining code outputs are prevalent, but the variety and novelty of task designs impede comparability across studies. The paper stresses the necessity for tasks aligned with theoretical frameworks and the exploration of psycho-physiological measures as promising tools for deeper insights into cognitive processes.

Figure 4: Comprehension measures used relative to the number of papers.

Implications and Future Directions

Theoretical and Practical Contributions

The mapping paper elucidates the lack of a unifying theory for code comprehension, suggesting a focus on foundational research to develop robust cognitive models. Such theoretical grounding would facilitate the design of more coherent and comparable studies.

Recommendations

• Establish clear definitions and operationalizations for code comprehension constructs to standardize research methodologies.
• Explore the impact of code comprehension on other constructs to unveil broader implications for software engineering practices.
• Enhance reporting standards for paper designs, ensuring detailed descriptions of participant demographics, tasks, and measures to support reproducibility and secondary analyses.

In conclusion, while significant strides have been made in understanding code comprehension, the paper identifies critical areas for future research endeavors, particularly in establishing theoretical underpinnings and exploring practical applications of code comprehension findings.

Conclusion

The paper provides a comprehensive overview of the state-of-the-art in code comprehension studies, offering valuable insights into common practices, methodological advancements, and persistent gaps. Future research should aim to address these gaps through theoretical development and methodological standardization, ultimately enhancing the field's contribution to effective software development practices.