-
Notifications
You must be signed in to change notification settings - Fork 13
/
Copy path06-release-engineering-analytics.Rmd
383 lines (310 loc) · 19.1 KB
/
06-release-engineering-analytics.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
# Release Engineering Analytics
## Motivation
Release engineering is a software engineering discipline
concerned with the development, implementation, and improvement of
processes to deploy high-quality software reliably and predictably [@dyck2015a].
The changes made by the developers of a software system
should eventually be integrated and deployed such that end users may benefit
from them.
In recent years, software industry has
adopted techniques to build infrastructure and pipeline which automate the
process of releasing software to an increasingly large degree.
These modern release engineering practices have resulted
in adoption of various other practices such as releasing new versions of a software system in
significantly shorter cycles.
Most of the developments in release engineering are industry-driven, with less research to support the developments.
Efforts to close this gap would be relevant both for practitioners as well as
researchers [@adams2016a].
On the one hand, claims and rationales are presented by the industry to justify
practices in release engineering, but these are often not empirically validated.
For this reason, research should aim to build an understanding of the actual
effects of release engineering practices on the software development process.
On the other hand, software engineering researchers need to be aware of modern
release engineering practices in order to account for them in their analyses.
Otherwise, their lack of familiarity with these practices will likely result
in biases in their study results.
This systematic literature review aims to provide an overview of
the software analytics research that has been conducted so far on modern
release engineering.
Its main purpose is to identify the apparent gap
between research and practice, in order to guide further research efforts.
### Research Questions
Contrary to what is regularly the case, advances in release engineering
practices are driven by industry, instead of scientific research.
Building on this idea, our questions are constructed to identify in which ways
existing modern release engineering practices should still be studied in
software analytics research.
Our review thus aims to answer the following questions.
- **RQ1:** _How is modern release engineering done in practice?_
This question aims to identify the so-called "state of the practice" in
release engineering.
We will summarize practices that have been adopted in modern release
engineering. In addition, we will identify the tools utilized to implement modern release engineering.
- **RQ2:** _What aspects of modern release engineering have been studied
in software analytics research so far?_
In order to answer this question, we investigate the practices that previous
empirical studies have focused on.
In doing so, we identify the associated costs and benefits
that have been found, and the analysis methods used.
- **RQ3:** _What aspects of modern release engineering make for relevant
study objects in future software analytics research?_
By answering this question, we aim to identify the gaps between practice and
research in release engineering.
This way, our intent is not only to guide but also to motivate future research.
## Research Protocol
We follow the procedure to conduct systematic reviews as proposed by Kitchenham @kitchenham2004procedures.
We set up strategies for searching, selecting and
quality-assessing studies.
Subsequently, we extracted data from the selected studies and
synthesized the answers to our research questions.
All the papers found were stored in a custom-built web-based tool for
conducting literature reviews.
The source code of this tool is published in a GitHub repository.
^[See https://github.com/jessetilro/research]
The tool was hosted on a virtual private server such that all retrieved
publications were stored centrally, accessible to all reviewers.
In order to save space in this chapter of the book,
we have omitted the full research protocol from this chapter.
The interested can find our research protocol in detail in
[Section 7.5](#appendix).
## Answers
In this section, we answer each research question presented in [Section 7.1.1](#research-questions-1).
### RQ1: Modern Release Engineering Practices
_How is modern release engineering done in practice?_
Adams et al. @adams2016a and Karvonen et al. @karvonen2017a have described release engineering
practices that are currently in use in the industry.
Adams et al. @adams2016a focused on modern release engineering,
while Karvonen et al. @karvonen2017a investigated agile release engineering,
which is a subset of all modern release engineering.
Both studies agree that modern release engineering consists
of the following components:
- **Rapid Releases (RR).**
In contrast with traditional release cycles, software systems following
Rapid Release cycles regularly push new releases to users in a regular schedule.
For example, FireFox releases a new version every six weeks.
- **DevOps.**
According to Dyck et al. @dyck2015a: "DevOps is an organizational approach
that stresses empathy and cross-functional collaboration
within and between teams (especially development and IT operations)
in software development organizations, in order to operate
resilient systems and accelerate delivery of changes."
- **Continuous Integration (CI).**
To quote Adams et al. @adams2016a: "[CI] refers to the activity of continuously polling the
[version control system] for new commits or merges,
checking these revisions out on dedicated build machines,
compiling them and running an initial set of tests to check for regressions."
- **Continuous Deployment or Continuous Delivery.**
When the CI tests pass, the code can be automatically deployed
to the production environment.
The difference between these terms is that continuous delivery
does not require that changes are automatically deployed,
but continuous deployment always automatically deploys changes.
However, the change might not yet be released,
because it can be hidden using feature toggles [@laukkanen2018a].
Besides these four components, Adams et al. @adams2016a also identified three other concepts
that are used in modern release engineering,
specifically to release software as often as possible
and thus enable continuous deployment or delivery:
- **Branching and Merging.**
There are several possible strategies of collaboration. Branching and merging is one of them and is most widely used.
Typically, merging must be done as often as possible
in order to avoid conflicts thereby enabling release of software as often as possible.
- **Build System.**
With the build configuration stored inside the project,
every developer (or automated tool) only needs to issue a single command
in order to build the project,
instead of manually having to configure the build process every time.
Building a software must be done in a consistent way,
such that each build produces the same result.
- **Infrastructure-as-Code.**
In the same alley of "storing configuration",
infrastructure-as-code means that the server (or virtual machine)
on which a software product is running
can also be automatically configured with code,
instead of having to configure each server manually.
Besides these seven components that are more technical, Poo-Caamaño @poo-caamano2016a has
identified that there are also social aspects to modern release engineering. Specifically, most
large software projects have a dedicated Release Team that will decide on the release strategies
and communicate them to others.
### RQ2: Studied Parts of Release Engineering
This section aims to answer to the question:
_What aspects of modern release engineering
have been studied in software analytics research so far?_
Over the years, the software industry has come up with innovative approaches to deliver new
features and fixes in a more efficient and faster manner. This has resulted in case studies
being done to assess the associated risk and cost factors, and what benefits certain
strategies can give.
Khomh et al. @khomh2015a have looked into the effects of switching from traditional to rapid
release cycles
in the case of Mozilla Firefox. The paper has concluded that users do not experience
significantly more post-release bugs, bugs are fixed faster, but that users experience bugs
earlier in the software execution. Mantyla et al. @mantyla2015a have also considered data from
Mozilla
Firefox and has examined the impact of release engineering on testing efforts. Observations of
the paper conclude that the rapid release cycle performs more test executions per day, but
these tests focus on a smaller subset of the test case corpus and that testing happens closer
to release and is more continuous. Da Costa et al. @da2014a has further zoomed into the integration of
addressed issues
and have considered data from Mozilla Firefox, as well as data from ArgoUML and Eclipse. The
paper found that addressed issues are usually delayed in a rapid release cycle and are often
excluded from releases.
Similar conclusions based on Mozilla Firefox were made by Da Costa et al. @da2016a,
who found that minor-traditional releases tend to have
less integration delay than major/minor-rapid releases.
Castelluccio et al. @castelluccio2017a examined the practice of *patch uplifting* in the
release management at
Mozilla Firefox where patches that fix critical issues, or implement high-value features are
often promoted directly from the development channel to a stabilization channel. The paper
evaluated the characteristics of patch uplift decisions and interviewed three Mozilla release
managers. The paper concluded that the majority of patch uplift decisions are made due to a
wrong functionality or crash. The specificity and code author of patches that are requested to
be uplifted are also a major factor for release managers.
In response to case studies being done on many prominent open source software projects,
Teixeira @teixeira2017a has described OpenStack's shift to a liberal six-month release cycle. As
this
is an ongoing study, the results given by the paper are preliminary and only observe the
process. OpenStack's release process can be considered as a hybrid of feature-based and
time-based releases. OpenStack encourages regular releases but also attempts to include new
features at each regular release.
Rather than focussing on topics such as issue and delays, Poo-Caamaño @poo-caamano2016a focusses
on the
communication in release engineering in the cases of GNOME and OpenStack. Through analyzing
over 2.5 years of communication, the paper has made a number of observations.
The paper found that developers tend to communicate through
blogs, bug trackers, conferences, and hackfests.
Another finding is that a release team is set to define
requirements, quality standards, and coordination through (direct) communication.
Although only the mailing lists of the projects were studied,
defined challenges include keeping everyone informed and engaged,
monitoring changes and setting priorities in cross-project coordination.
Laukkanen et al. @laukkanen2018a have described what effects modern release engineering
have on software with different organizational contexts.
This study specifically focusses on continuous deployment practices.
The paper has found that high internal quality standards combined with the
large distributed organizational context of large corporations slowed the verification process
down and therefore had a negative impact on release capability. However, in small
corporations, the lack of internal verification measures due to a lack of resources was
mitigated by code review, disciplined CI and external verification by customers in customer
environments. More about the factors that can play a role is addressed by Rodríguez et al.
@rodriguez2017a,
where an overview of contributing factors in continuous deployment are defined and categorized
based on literature between 2001 and 2014.
As rapid release cycles and continuous deployment are topics that are new and emerging, not
enough research has been done to generalize any conclusions that are made in the case studies
discussed in this section. This is why all the empirical studies in this survey have one major
sidenote in common: more case studies are needed. Open challenges such as these will be
discussed in the next section.
### RQ3: Future Research
This section aims to answer the question:
_What aspects of modern release engineering make for
relevant study objects in future software analytics research?_
#### General Suggestions
The body of literature that we analyzed for this survey mostly comprised case
studies that employed quantitative analysis methods.
From these studies, interesting conclusions have been drawn about the effects of
release engineering practices on software development processes in specific
contexts.
However, the generalizability of the findings in these case studies is very
limited.
Therefore, in general many studies suggest that future research efforts focus on
performing additional case studies, both to verify existing findings and to
study new relationships and new contexts [@karvonen2017a; @teixeira2017a;
@khomh2015a; @claes2017a; @laukkanen2018a; @adams2016a; @castelluccio2017a].
It also seems worthwhile to triangulate findings by complementing data analyses
with other quantitative (e.g. a survey) or qualitative (e.g. an interview)
methods [@karvonen2017a].
Finally, additional literature reviews will allow researchers to keep an
overview of the most recent developments and findings in the area of release
engineering [@rodriguez2017a; @laukkanen2018a].
Apart from verifying results, it might be worthwhile to leverage them by
constructing analysis tools for practitioners.
For example, Castelluccio et al. @castelluccio2017a suggest exploring possibilities to leverage
their research by building classifiers capable of automatically assessing the
risk associated with patch uplift candidates and recommend patches that can be
uplifted safely.
Also, companies seem to be struggling with the adoption of continuous delivery
and deployment, so a checklist for analyzing readiness for these practices
might be developed [@karvonen2017a].
The review by Karvonen et al. @karvonen2017a makes a number of general suggestions for future
research.
In particular, there should be more attention to comprehensively reporting how
practices are implemented and in which context they are embedded, instead of
just stating that they are used.
Also, the viewpoints of different stakeholders other than developers can be
taken into account.
For example, the customer perceptions regarding the adoption of a certain
practice can be investigated.
#### Directions for Specific Practices
**Rapid Releases**
As established, rapid releases are a prevalent topic in current research on
modern release engineering.
However, it will be useful to verify these results given the fact that this
research mainly involves case studies (most of which are only concerned with
Mozilla Firefox due to the availability of data).
To this end, there are opportunities to further investigate the effects of
switching to rapid releases on:
- code integration [@da2014a; @da2016a; @souza2015a; @castelluccio2017a],
- testing efforts [@mantyla2015a],
- software quality [@khomh2015a; @khomh2012a],
- (library) adoption [@fujibayashi2017a]
- time pressure and work patterns [@claes2017a].
**DevOps**
When it comes to DevOps, future research is needed to refine its definition such
that it is uniform and valid for many situations [@dyck2015a]. According to
Karvonen et al. @karvonen2017a, it seems that the goals in DevOps are congruent with those
in release engineering, and future research on this topic is therefore highly
relevant in order to study modern release engineering.
**Continuous Delivery / Deployment**
Research on continuous deployment seems to be still in its infancy, therefore Rodríguez et al.
@rodriguez2017a have suggested a significant number of different concrete
opportunities for future research.
In general, they conclude that the topic needs an increase in the number and
rigor of empirical studies, and thus it presents opportunities for software
analytics research.
In a systematic literature review, Laukkanen et al. @laukkanen2017a identified 40 problems,
28 causal relationships and 29 solutions related to the adoption of continuous
delivery.
These problems and solutions can be studied further to deepen the
understanding of the nature of these problems and how to apply their solutions.
Some of the problems that are more of a human or organizational nature might
be involved with a broader spectrum of changes, so it should also be
investigated to what extent these problems are specific to continuous delivery.
**Continuous Integration / Build system**
One of the main issues that seems to be obstructing organizations in adopting
modern release engineering practices is build design [@laukkanen2017a]. In a
case study by Laukkanen et al. @laukkanen2018a, it was found that a complex automated build and
integration system led to a more undisciplined one, which in turn slowed down
the verification and release processes. Therefore, future research might
investigate how developers can make their builds more maintainable and of
higher quality, how anti-patterns in the design of the build can be refactored,
and how continuous integration can be made faster and more energy efficient
[@adams2016a].
## Conclusion
In this literature survey,
we have provided an answer to the following three research questions:
- **RQ1:** _How is modern release engineering done in practice?_
We found that there are six important technical aspects to
modern release engineering: Rapid Releases, DevOps, Continuous Integration,
Continuous Deployment, Branching and Merging, Build Configuration and
Infrastructure-as-code.
The most important social aspect of modern release engineering is communication.
- **RQ2:** _What aspects of modern release engineering have been studied
in software analytics research so far?_
At this point in time, case studies have mainly focussed on
the resulting factors of switching from a traditional release cycle
to a rapid release cycle, and what effects this has in various
organizational contexts.
As all included studies suggest, more empirical studies are needed to be
able to make general conclusions in the novel field of release engineering.
- **RQ3:** _What aspects of modern release engineering make for relevant
study objects in future software analytics research?_
In general, more empirical research is required to validate and generalize
the results of many previous case studies within the field of release
engineering.
In addition to performing case studies involving quantitative analyses,
it may be beneficial to triangulate results using various research methods.
Also, future research should more comprehensively describe how practices
are implemented, and consider different stakeholders.
For each practice, future research is suggested on the one hand to further
investigate their effects on the development process, and on the other hand
to investigate problems involved with their adoption.