Uncovering the Benefits and Challenges of Continuous Integration Practices

In 2006, Fowler and Foemmel defined ten core Continuous Integration (CI) practices that could increase the speed of software development feedback cycles and improve software quality. Since then, these practices have been widely adopted by industry and subsequent research has shown they improve software quality. However, there is poor understanding of how organizations implement these practices, of the benefits developers perceive they bring, and of the challenges developers and organizations experience in implementing them. In this paper, we discuss a multiple-case study of three small- to medium-sized companies using the recommended suite of ten CI practices. Using interviews and activity log mining, we learned that these practices are broadly implemented but how they are implemented varies depending on their perceived benefits, the context of the project, and the CI tools used by the organization. We also discovered that CI practices can create new constraints on the software process that hurt feedback cycle time. For researchers, we show that how CI is implemented varies, and thus studying CI (for example, using data mining) requires understanding these differences as important context for research studies. For practitioners, our findings reveal in-depth insights on the possible benefits and challenges from using the ten practices, and how project context matters.