This essay provides an overview of randomized controlled trials (RCTs) and their meteoric rise. Then the author discusses key limitations to the use of RCTs, from a theoretical and methodological lens and provides a country-study example.
In recent times, randomized controlled trials (RCTs) as a method of impact evaluation of the effectiveness of development interventions have sparked enthusiasm and optimism among a wide range of stakeholders. Evidence from randomized evaluations is increasingly being used to understand and address poverty-related problems. To that extent, advocates of RCTs labeled the "randomistas" as proffer RCTs the "gold standard" for developing effective poverty reduction policies. Gold standard, in this sense, being the best and most accurate method (without restriction) to designing development interventions and assessing what works and what does not.
Table of contents
1. INTRODUCTION
2.0 A NEW WAY OF DOING ECONOMICS: UNDERSTANDING RANDOMISED 2 CONTROLLED TRIALS (RCTS) AND THEIR PROLIFERATION
2.1 HOW DOES RANDOMISATION WORKS?
2.2 PROLIFERATION OF RANDOMISED CONTROLLED TRIALS (RCTS) IN CONTEMPORARY 3 DEVELOPMENT
3.0 ASSESSING RANDOMISED CONTROLLED TRIALS (RCTS): EXPLANATORY LIMITATIONS SURPASS SMALL-SCALE FINDINGS
3.1 CONCEPTUAL PROBLEM
3.2 METHODOLOGICAL SHORTCOMINGS
4. SCALING UP WHAT WORKS: A CASE STUDY OF KENYA
5. CONCLUSION
APPENDIX
References
1. INTRODUCTION
In recent times, randomized controlled trials (RCTs) as a method of impact evaluation of the effectiveness of development interventions have sparked enthusiasm and optimism among a wide range of stakeholders (e.g. governments, donors, Academia and NGOs). Evidence from randomized evaluations is increasingly being used to understand and address poverty-related problems (Deaton & Cartwright, 2018). To that extent, advocates of RCTs labeled the 'randomistas' as proffer RCTs the 'gold standard' for developing effective poverty reduction policies. Gold standard, in this sense, being the best and most accurate method (without restriction) to designing development interventions and assessing what works and what does not (Banerjee, 2006; Duflo, 2017).
The increasing popularity of randomized controlled trials (RCTs) in the field of development, however, has raised concerns. Economists such as Angus Deaton, Dani Rodrik, Lant Pritchett and Martin Ravallion, to name a few, remain critical of their use in economic experiments and in reducing poverty. RCTs have been criticised for being unethical, for producing results with weak external validity, for being too expensive and for its poor contextualisation (Deaton & Cartwright, 2018; Kabeer, 2019; Kvangraven, 2020; Ravallion, 2018; Reddy, 2012)
Whilst there is some truth in the critiques, this does not overturn RCTs in general, as the contributions of the randomistas and their evidence have meaningfully improved the way development work is implemented in practices. Rather, it points to the limitations in the method that RCTs proponents and users should be aware of. Dubbing RCTs as the 'gold standard' is overly ambitious, the method alone is "not capable of significantly alleviating poverty, let alone eliminating it" (Akram-Lodhi, 2014, p.429). As such, RCTs such be used in complementary with other nonexperimental methods and within a theory-based framework, whenever possible (Binci & Jasper, 2020). Section 2 of this essay provides an overview of RCTs and their meteoric rise. Section 3 discusses key limitations to the use of RCTs, from a theoretical and methodological lens. Section 4 provides a country-study example. Section 5 concludes this essay.
2.0 A NEW WAY OF DOING ECONOMICS: UNDERSTANDING RANDOMISED CONTROLLED TRIALS (RCTS) AND THEIR PROLIFERATION
The World Bank's Handbook on Poverty and Inequality (2009) defined poverty as a deprivation in well-being. In particular, a World Bank report (2015) maintained that "poverty is not simply a shortfall of money, it includes the constant, day-to-day hard choices associated with poverty that tax an individual's bandwidth or mental resources'^ p.81). This cognitive burden, thus, makes it difficult for the poor to think deliberately, leading them to make non-optimal choices that perpetuate poverty (Mani et al., 2013).
The understanding of behaviours is increasingly being used to find new ways to solve poverty problem and to provide new perceptions into key questions about why the poor remain poor and why people choose and behave in certain ways, even after the problems of provision, access, and pricing have been resolved (Banerjee & Mullainathan, 2008; Datta & Mullainathan, 2014; Mullainathan, 2005). Randomised Controlled Trials (RCTs) draw greatly on the precepts of behavioural understanding, breaking away from the standard economic (rationality) theory (Fine & Santos, 2016; Stevano, 2019). This section provides an overview of how RCTS works, and their meteoric rise in the field of contemporary development economics.
2.1 HOW DOES RANDOMISATION WORKS?
Randomised controlled trials (RCTs) are experiments designed to "compare the outcome of an intervention with what would have happened had the intervention not taken place, in order to measure its net impact" (Bedecarrats et al., 2017, p. 737). Under RCTs experiment, two samples are drawn at random from a target population. One group is given the intervention (Treatment) and the other group is not (Control). The two groups are then surveyed, before and after the project, to compare the effects of the intervention. Any differences observed between the Treatment and Control group is ascribed to the intervention, within a certain margin of error, having controlled for all relevant variables (Akram-Lodhi, 2014).
The results of RCTs can then be used to formally establish the true effect of a causal variable upon an outcome of interest (Bedecarrats et al., 2017). Using RCTs, the randomistas claim to provide "an unbiased estimate of the average impacts of a programme, by measuring the difference in the mean outcomes reported by treatment and control groups,its average treatment effects" (Kabeer, 2019, p.199).
2.2 PROLIFERATION OF RANDOMISED CONTROLLED TRIALS (RCTS) IN CONTEMPORARY DEVELOPMENT
The increasing use of RCTs stem from "its mix of simplicity, mathematical rigour; intense whirl of communication and advocacy, and academic and financial returns, that appeal to the interest and preference of many in the academic world and the donor community" (Bedecarrats et al., 2017, p.735). RCTs focus on small, narrow and simple questions that are amenable to randomised trials like 'do textbooks improve students' outcomes?, as such, the method is able to distinguish between simple correlations, causal contribution claims (i.e. factors that might have impacted measured outputs and outcomes) and attribution claims (i.e. the detected impact on indicators of interest attributed to an intervention) (Binci & Jasper, 2020). This lend appeal to a range of stakeholders such as government and funders, who wants valuable and robust evidence on the effectiveness of programmes, to inform and guide their decision making.
The Abdul Latif Jameel Poverty Action Lab (J-PAL), co-founded by two of the 2019 Nobel Laureates, Abhijit Banerjee and Esther Duflo also played an influential role in the proliferation of RCTS. Toward their mission of reducing poverty by providing scientific evidence to guide policy, J-PAL only does RCTs. For them, "RCTs are not just top of the menu of approved methods, nothing else is on the menu" (Ravallion, 2018, p.11). The evidence-to-policy page of J-PAL website claim more than 400 million people have been impacted by programs that were scaled up after evaluation by J-PAL affiliated researchers.
3.0 ASSESSING RANDOMISED CONTROLLED TRIALS (RCTS): EXPLANATORY LIMITATIONS SURPASS SMALL-SCALE FINDINGS
Randomised controlled trials (RCTs) have an important role in designing effective poverty reduction programs, however, an absolute preference for RCTs as the "gold standard" against other methods is questionable. This section discusses the conceptual and methodological limitations of RCTs method.
3.1 CONCEPTUAL PROBLEM
There are restrictions on the kind of questions posed and tackled by randomised controlled trials (RCTs). They only answer small, relatively simple, and easily actionable questions like whether two schoolteachers in the classroom are much better than one, or why undernourished people do not spend more on nourishing food (see, e.g., Ravallion, 2012; Reddy, 2012). RCTs do not attempt to solve the big, important economic and structural problems that prevail most developing countries such as trade imbalances, scarce public provision of social services and lack of productive employment opportunities (Binci & Jasper,2020; Kvangraven, 2020; Stevano, 2019).
The very fact that RCTs only focus on specific questions is a problem in itself, since economic research should look at bigger and more fundamental questions. As Akram-Lodhi (2014) put:
RCTs far too commonly deal with the symptoms of a development problem rather than trying to understand the underlying causes of that problem. Yet the mechanics of individual and household choice cannot be seen in isolation from the overall socio-economic structure, which must be investigated, (p. 428)
Small questions may be important, but they should not take priority over macro-level issues that causes high degrees of poverty and inequality to begin with. As emphasised by Ravallion (2012), the actual development programs done by governments to fight poverty addresses the big issues (like trade and industrial policies, road building, poor-area development programs etc.), and they do need to know whether they work or not. Because RCTs deal with the responses of individuals or households to a well-defined treatment, it is unlikely to address the big structural problems using randomizations, for they are multifaceted global issues that cannot be narrowed to a single unit (Kvangraven, 2020).
3.2 METHODOLOGICAL SHORTCOMINGS
Randomised controlled trials (RCTs) have been praised for its reliability of results as opposed to other methodology in development economics (Bedecarrats et al., 2017). How credible is the result of RCTs outside its original context (i.e. external validity) is, however, questionable. Deaton and Cartwright (2018) strongly contest against the popular notion that the "average treatment effect calculated from an RCT is automatically reliable" (p.2). They argued that RCTs estimates might not be precise, even if true, such truth only holds within the trial sample. Hence, their results cannot be used for predictions outside the original context. For that, there is need for more structure, more prior information, and a good theory of what makes treatment effects vary outside the original context, the type of assumptions and knowledge RCTs claim not to use.
Randomistas may be swift to reply that such limitation is not peculiar to RCTs. Of course, no approach can by default claim to be applicable to any context or population, however, RCTs have a disadvantage over other research methods that use a broader range of prior information to understand the 'why' behind results (Deaton & Cartwright, 2018).
In addition, the limited predictive explanatory power of RTCs and focus on trial and error mechanisms makes them money and time costly.
[...]
-
Upload your own papers! Earn money and win an iPhone X. -
Upload your own papers! Earn money and win an iPhone X. -
Upload your own papers! Earn money and win an iPhone X. -
Upload your own papers! Earn money and win an iPhone X. -
Upload your own papers! Earn money and win an iPhone X. -
Upload your own papers! Earn money and win an iPhone X.