A little bit of history

Did you know that you are subject to the evidence-based practice (EBP) whenever you visit a doctor? Have you ever wondered why you are (usually =) so confident that your physician has the right treatment for you? Implicitly, it seems that all of us assume that the treatment is selected based on one or more (factual) reasons for believing that it will have positive effects on your illness. Not by chance, this is precisely the definition of evidence¹.

Isn’t that interesting? We expect that physicians base their practice on evidence, but we in the software community seem to prefer to argue about whether a practice brings happiness or a framework provides an elegant solution for a problem. Not that these aspects aren’t important, but you got the point. I have never met a software developer (outside the academic circle) that is aware of the EBP. If you are one, please leave a comment! I would be glad to know who (and how many) you are.

The notion of EBP dates from the beginning of the 1990s in the Medicine discipline. In 1990, Dr. Gordon Guyatt introduced a new concept called “Scientific Medicine” to describe a new method of teaching medicine at the bedside, using the critical appraisal techniques from Dr. David Sackett. One year later the term “Evidence-Based Medicine” (EBM) was coined in the ACP Journal Club Editorial². The basic idea of EBM was to bring more certainty to clinical decision making. Before EBM, “clinical practice was historically viewed as the ‘art of medicine.’ Expert opinion, experience, and authoritarian judgment were the foundation for decision-making. The use of scientific methodology, as in biomedical research, and statistical analysis, as in epidemiology, were rare in the world of medicine.”³ This scenario looks familiar, eh? Any resemblance to our current situation is purely coincidental.

Since its proposal, EBM received huge attention. The term was then generalized from EBM to EBP. And several fields have adopted EBP aiming at improving its decision-making process such as Administration, Nursing, Marketing, and, of course, Software Engineering. In our case, EBP was brought in 2004 by Barbara Kitchenham and her colleagues in the International Conference on Software Engineering⁴. Still, even though this was the first mention of EBP in Software Engineering, which was then denominated Evidence-Based Sofware Engineering (EBSE), the field promotes the evidence (through experimentation) since 1978. The general research topic under which experimentation and evidence are investigated is called Empirical Software Engineering.

Evidence-Based Software Engineering (in the view of researchers)

Ok, so now that you know the origin of EBP and what it is about, it is time to understand how EBP can help our field to get out of the “fashion-oriented industry” to a real engineering field. Before presenting my view about how EBP should be in software engineering, I’ll present it “as is” (in the view of researchers). Based on the definition from Medicine, the EBSE was defined as follows⁴:

To provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision-making process regarding the development and maintenance of software.

To achieve this goal, researchers from Empirical Software Engineering operationalized EBSE in five steps⁵:

1. Convert a relevant problem or information need into an answerable question;
2. Search the literature for the best available evidence to answer the question;
3. Critically appraise the evidence for its validity, impact, and applicability;
4. Integrate the appraised evidence with practical experience and the customer’s values and circumstances to make decisions about practice;
5. Evaluate performance and seek ways to improve it.

The EBP is about asking the right question (step 1) — to able to search for the evidence available. Thus, instead of looking for the “hype of the moment”, EBP should enable us to seek for the reasons to believe that a software development technology will provide the support we expect from it. But how can you find the evidence?

I’ll try to be succinct here since the steps do not make this clear. They only hint that you will find the answer for your problem in the literature. However, surveying the literature is a hard task. There are a huge amount of scientific articles published, which must be screened to obtain the answer. To perform this orderly, systematic review⁶ is proposed as a procedure to translate the “right question” into a search string and the criteria to appraise the evidence (steps 2 and 3). Fortunately, several topics in software engineering have been targeted of systematic reviews (up until 2010 there were 67 reviews covering 24 topics⁷). After that you are on your own (steps 4 and 5), it is expected that you understand the papers and can extract the necessary information to integrate the evidence with your practical experience. As one can see, this is not an easy task as well. Below, I’ll explore this and other aspects regarding the researcher view of EBSE.

Why EBSE isn’t widely adopted by software engineers?

Based on the above definitions, I can’t blame you for not knowing what EBSE is. I could list a number of arguments for that, but there is one which in my view is sufficient to explain the situation. EBSE states that you must search the best evidence from research, which “for a good understander”⁸ means that software engineers do not know how they can provide evidence for technologies. Yes, it says that you can integrate it with practical experience, but it is clear that evidence (from research) is what is important. After all, the practice is called Evidence-Based Software Engineering. Strictly speaking, the EBSE suggests an artificial hierarchy between researchers and software engineers, which obviously is not going to be accepted by the latter. Why would software engineers have to accept prescriptions regarding how we should work, even in the form of evidence?

As if that was not enough, precisely because most researchers commend this segregation between who investigate and who apply the findings, it is common to find results that are not directly applicable or are not relevant to the practice. Not to mention that when they are applicable, the results are usually “buried” in a highly technical propositional text written with the maximum rigor in mind, but not necessarily readability.

In addition to these issues, which I consider the most salient, there are several other discussions regarding the utility of EBP. For instance, some issues that are directly translatable from the critics of EBM: “some claim that it transforms the complex process of clinical decision-making, which includes data gathering, years of medical knowledge, experience, and astute intuition into an algorithmic exercise that is not individualized for specific clinical scenarios and therefore subject to error in patient care”³.

How can we change that?

The EBP is clearly not a panacea. Still, apart from the issues discussed, I hope that you are convinced that it is imperative we stop creating new technologies without any evidence regarding its utility and potential benefits. Hence, it is also urgent that we find the means to build a body of evidence that is readily available for software engineers. In my view, there is three main areas that we must act upon: (i) software engineers producing evidence from their practice, (ii) a description format⁹ for evidence, and (iii) an infrastructure to support searching and managing the available evidence. With these areas addressed, the EBP culture can flourish, and our community will benefit from informed decisions which will contribute to increasing our focus on software development technologies that (really) support our work.

In this blog, I expect to contribute to all these areas. Regarding the item (i), it is my intention to make this blog a reference for how you can produce (and use) evidence. For the second item, I’ll present evidence published in scientific journals, concentrating — and translating if necessary — on the relevant aspects to inform decisions on practice. By the way, if you have any theme you are interested in, please leave a comment. I also intend to discuss the description format itself. The last item is a community endeavor; I’m optimistic that we can start this together.