Many of you would have been aware of the increasing use of randomised evaluations in Social Science research and for public policy reasons. Taking an epistemological look, I give a robust argument on why the claims of randomised evaluations actually evade the problem of induction. Hope to get your thoughts. Thanks in advance !

1. Introduction

The usage of randomised evaluations in social inquiry has been recent and responses to them have been wide ranging. Some have described it as the “gold standard” in empirical research, (Duflo, Glennerster,&Kremer, 2006) while others though have been more critical of their value in making predictions. (Deaton, 2009)

Randomised evaluations[1] (REs) seek to make predictions on the impact of an intervention, when it is attempted in a new situation. REs work by first determining the impact of the intervention. Subsequently, for the new situation it is expected that the impact would be similar.

To determine an intervention’s impact, numerous subjects are selected and randomly assigned to either the treatment or control group. Intervention being investigated is introduced for the treatment group. The control group serves as a counter factual and no intervention is introduced. Then, after a certain period of time, the mean impact of subjects in both groups is evaluated. Given that all subjects were equivalent before the study, any difference in results between the treatment and control groups is attributed to the intervention.

 

 REs are increasingly being used to justify knowledge claims in Developmental Economics (Ester Duflo, 2006). Hence, it would be useful for us to examine their epistemological status.

 

In this paper, I will specifically focus on how the problem of induction poses a challenge to external claims of REs [2] being considered knowledge. While accepting that no viable solution exists to the problem of induction faced by RE’s external claims, I will argue that we ought to consider “inductive validity” rather than deductive validity as the standard to be met. By further elaborating what would constitute an inductively valid inference, I will show that the criteria can be met by external claims and hence conclude that the problem of induction does not undermine their epistemic status. 

2. Claims of RE and Problem of Induction

Typically in a RE, there are two types of knowledge claims made- Internal and external. Internal claims pertain to the observed impact of the intervention. For instance, in a RE conducted to find the impact of providing subsidised meals on school attendance, the internal claim was that the intervention increased attendance rate by 30 percent. (Christel Vermeersch, 2004) External claims, on other hand, are predictions of impact. For the same RE, the external claim would be a prediction of comparable impact when the intervention is administered in another situation, say a neighbouring school. 

 Much of RE’s epistemic superiority over other empirical methods has been credited to its ability to generate highly justified internal claims. Unlike a standard empirical study, in a RE, a control group is always constructed (Ester Duflo, 2006). The control group shows accurately what the impact would have been if not for the intervention and thus any difference between the treatment and control is attributed to the intervention alone. Also, the samples are randomized, reducing selective bias. Internal claims are not undermined by the problem of induction as there is no element of projection within them. 

 

The high internal validity however does not mean that external claims do not face the problem of induction. External claims are justified inductively, based on the assumption of “uniformity of nature”. However, this assumption is presumed without its validity being proven (Hacking, 2001) Hence, the question of whether external claims are justified arises.

Solving or evading the problem of induction is important. The primary purpose of REs is to determine high impact interventions and scale-them up. Thus if an appropriate response to the problem of induction is lacking, then belief in the external claim is thoroughly undermined.

Any solution to the problem of induction, involves proof that assumption of “”uniformity of nature” holds. Doing so warrants “deductive validity”, allowing the external claim to be considered knowledge. An evasion, on other hand, responds to the problem by asserting that the “uniformity of nature” assumption need not be proven valid for an inductively reasoned belief to be accepted as knowledge. (Hacking, 2001) Both are possible responses that would be considered, developed and evaluated next.

3.  Solution to the Problem of Induction

Goodman (2002) argues that the problem of induction is intractable considering that it demands distinguishing predictions “antecedently between true and false” and this task is akin to demanding “prevision rather than philosophical explanation”. He asserts that the assumption of the uniformity of nature cannot be proven in all fields employing inductive reasoning. 

This is clearly not the case. With John Foster’s (2004) Nomological Explanatory Solution (NES) it is possible to validate the assumption in Natural Science studies.  However, as I will show subsequently, such solutions are indeed ineffective for in case of RE’s external claims.

The key ideas of NES can be summarised as follows

1)    If there is consistent regularity in observations , then it cannot be a mere coincidence

2)    Inference to the best explanation is used to elucidate the regularity 

3)    The explanation “involves the postulation of some law or set of laws of nature” , which have to be considered as naturally necessitating the observation

4)    The explanatory “law” entails the weaker claim, that it will hold in the future as well

To illustrate how the NES works, consider the case of graphite always conducting electricity when charge is applied. The observation that graphite consistently conducts electricity demands an explanation. In response, we could explain this phenomenon by arguing that graphite has delocalised electrons thereby enabling it to carry electricity. The explanation necessitates that graphite conducts electricity, whenever charge is applied. From this explanatory law, the weaker claim that “graphite will be an electric conductor in the future “is entailed as well. When NES is applied, there are no extrapolative steps and the problem of induction appears to be solved.

While NES is effective in solving the problem of induction for Natural Science claims, the same cannot be said of RE’s external claims. In the case of REs, enumerative induction is used to justify external claims. No rational explanation is offered for the difference in average impact between the treatment and control group. Hence, the belief in the uniformity of nature is speculative.

In contrast, inference to the best explanation is used to account for past regularities in the Natural Sciences .Forming explanations  to elucidate results obtained is a central feature of scientific inquiry (G.Hempel, 2001) .This qualifies NES to be an adequate solution for the problem of induction for Natural Sciences.

Even if efforts are undertaken to offer rational explanation for the impact changes observed, NES would still be ineffective. NES is only effective if results observed are a “product of natural necessity”. Otherwise the claim, that future occurrences would be similar, involves extrapolation and the external claim won’t be valid.

It is hard to argue that human behaviour is a complete product of natural necessity for it involves eliciting a specific action from human beings, who have free will. Causes might “incline but not necessitate” human action. (Flew, 1995)

To illustrate this better, consider the RE, where teachers were monitored using cameras and higher teacher attendance was observed (Hanna, 2005). A possible explanation for the results could be that teachers attend classes more regularly when they are monitored so as to maintain a positive image. Even if we take this explanation to be true, given any situation humans would always have some degree of freedom to choose how they act. In this case it is possible for some teachers to willingly decide not to attend work regularly despite knowing that their reputation would be sullied. Thus, monitoring teachers might induce but certainly not necessitate the expected action. This means that the assumption that nature will be uniform has not been proven. Thus, deductive validity is still lacking.

Compare this with the Natural Sciences where the explanatory laws invariably “require” that future occurrences be similar to past results. In the graphite example, it is inconceivable of graphite not conducting electricity when charge is applied. Hence, projections are deductively valid.

 Hence, it is incorrect for Goodman to claim that solving the problem of induction is intractable in the case of all fields. I have demonstrated that NES effectively solves the problem in case of the Natural Science field. However, the problem does indeed seem[3] unsolvable for REs.

4. Evading the problem of induction

A solution to the problem of induction faced by REs does seem inconceivable. However, just on this account we cannot conclude that belief in the external claim is undermined by the problem of induction. As established earlier in Section 2, in addition to considering the possibility of a solution to the problem, we need to consider whether the problem can be evaded or not.

Goodman (2002) suggests a compelling evasion by arguing that it is inappropriate for inductive inference to strive towards attaining deductive validity. He proposes that they should instead be judged on whether they attain “inductive validity”. Goodman’s argument is convincing as he distinguishes between inductive and deductive inferences and it is reasonable to have different epistemic standards for both types of inference.

This however raises the question of what criteria need to be fulfilled for an inference to be considered “inductively valid”. Strength of evidence is certainly a factor. We would be more justified in believing an inductive inference when it is backed up with strong evidence. By strong evidence I mean statistically significant evidence, which ensures that results obtained are not due to chance.

However as Goodman rightly points out, not all inferences are considered “inductively valid” despite strong evidence. For instance, assuming that there is a strong positive correlation between the number of Google searches of a country and the country’s prosperity, we would not be rationally justified to predict that a country will become more prosperous because it attempts to boost the number of searches. However, if there is a positive correlation of same magnitude between per capita spending on education and country’s level of prosperity, we would be considerably more justified to predict that when a country boost’s spending in education than its level of prosperity would increase.

The differences in treatment of both the inductive statements can be attributed to the former being an accidental inference and latter being a law-like inference. In a law-like inference, we can propose a justified explanation for why the causal factor necessitates, even if not completely, the result. Thus expectations of similar results are justified. In the earlier example of education spending, it is reasonable to believe that increased spending leads to a better skilled work force hence enhancing productivity. However there is no convincing logical explanation for the link between the number of searches of the country in Google and its prosperity level. This leads us to term the former statement as being an accidental inference and hence it cannot be considered “inductively valid”.

Thus far, we have identified two factors for an inference to be considered inductively valid. Firstly it must have strong evidential support and secondly it must be law-like. External claims of REs meet both these criteria.

Some might argue that REs must provide rational explanation to elucidate the relationship between the intervention and the result, so as to prevent any accidental inference. I, however, contend otherwise. For a law-like statement what ought to be proven is that the intervention cause the result being observed. The mutual presence of the causal agent and result must not be a mere accident. In the case of RE, any difference in results is attributed to the intervention being studied alone as all other relevant factors are kept constant.   Hence the intervention must have resulted in the differences in the result. Thus, it cannot be accidental.

To clarify this point, consider the RE conducted to establish the relationship between providing subsidised meals and students’ attendance rate in rural villages (Christel Vermeersch, 2004). From a set of similar schools, some schools were randomly selected, to offer a subsidised food service. Whereas in the control group, there was no such intervention. After 24 months, it was established that the students’ attendance rate for the schools in the treatment group was much higher than schools in the control group. Assuming all other parameters had been kept constant, the only difference between schools in both groups is the presence of intervention. Hence, we can conclude that the intervention must have caused the varying impact. Thus, any correlation cannot be an accident and RE claims are indeed law-like.

 In addition to being law-like, RE claims are also backed up by statistically significant results. All results are subjected to rigorous statistical testing and are only accepted if the probability of them arising due to chance is between 5 to 10 percent. (Esther Duflo, 2006). Thus REs are reasonably accurate and there is strong justification for expecting similar results. 

REs are limited to establishing that the intervention can cause the results obtained. For a new situation, they cannot prove that introduction of the intervention alone would be sufficient to expect similar results. This is because there might be unidentified confounding factors that also have to be jointly present to obtain similar results. For example, for the RE on school attendance rate, a factor which has to be jointly necessary might be close proximity of the school to the villages.  REs cannot identify such factors. So, some might contend that external claims are inductively invalid since we cannot confirm the presence of such confounding factors.

 However such an argument fails to take into account there are numerous subjects being tested in a RE. Given that allocation of the subjects into experimental and control groups is random, the presence of the necessary confounding variable would have occurred randomly in it as well. The results obtained are hence an aggregation of subjects with and without the confounding variable. Considering the probability of the confounding variable being present in the new situation is similar to that of the subjects being tested, the average impact would still be a good estimate. Hence there is good justification for external claims. 

In Section 3, I discussed about how the presence of human free will poses a challenge for RE’s external claims to achieve deductive validity. Some might extend that argument by asserting that inductive validity is also impossible since there is no guarantee that individuals would behave in a similar manner even same conditions are imposed.

 Such an argument commits the fallacy of composition by assuming that knowledge of how a group acts in response to an intervention is contingent on knowing how individual agents act. Admittedly, REs are limited in their ability to predict an intervention’s impact on an individual. However, this does not imply that external claims are not justified when making prediction on large groups.

This idea can be substantiated by invoking David Stove’s “Law Of Large Numbers” argument (Stove, 1986). Stove argues that we are justified to expect results to approximate to the actual probability when a large number of cases are sampled.

To illustrate this, let’s assume that a RE has determined that the chance of a teacher attending school because of a monitoring scheme being implemented is 80%[4]. The pre-determined chance is the same regardless of whether the number of subjects in the new projected situation is large or small. However, when the same intervention is attempted, we are more justified in believing that the attendance rate of teachers would be approximately 80% when the number of subjects in the new situation is large. For example, consider two schools with 100 and 5 teachers each. When the monitoring scheme is implemented in both schools, we are more justified to claim that attendance rate would be 80 out 100 rather 4 out of 5.  This illustration establishes that for external claims to attain inductive validity the number of subjects in the new projected situation should be high.   

5. Conclusion

The problem of induction arguably poses the most significant challenge to the epistemic status of REs’ external claims. To respond to the problem, the possibility of either solving or evading the problem was considered.

Through applying John Foster’s NES, I demonstrated that the problem of induction is not intractable in all fields as Goodman claimed. The problem of induction can indeed be solved in the case of Natural Science predictions. However, RE’s external claims pertain to complex, indeterminable human behaviour; as such it is impossible to validate the assumption of “uniformity of nature”. 

Lack of an effective solution does not mean that the epistemic status of REs’ external claims is undermined though. Using Goodman’s argument for evasion of the problem, I highlighted that it is only reasonable to expect inductive statements to attain inductive validity rather than deductive validity. This then raised a question of what constituted an inductively valid inference. 

To this, I proposed that inferences must be law-like and have strong evidential support. Also, we must note that REs can only provide an estimation of a trend for a large group rather than determine how individuals would exactly behave in response to an intervention. Hence, an additional criterion of the external claim being a prediction of impact over a large group also was also included.

The first two criteria of inferences being law-like and having strong evidential support are always met in an external claim because of the very methodology of REs. Whether impact predictions based on RE results, are made for large groups or not is determined by the user of the RE. Fulfilling this criterion is certainly feasible.  

Hence, it is possible for RE’s external claims to be inductively valid and thus they can successfully evade the problem of induction. This means that the problem of induction does not undermine the epistemic status of RE’s external claims. To conclude that external claims are knowledge would be however premature as other epistemic challenges have to be considered as well. But it does mean that external claims can still possibly be regarded as knowledge despite the problem of induction.

---

[1] Randomised evaluations have been used in many fields of Economics. However, in this paper, the term RE can be taken to exclusively mean randomised evaluations used in Developmental Economics.

[2]  The differences between internal and external claims will be elaborated in Section 2.

[3] We have only demonstrated that NES is ineffective in solving the problem of induction. There might be other possible solution which might possibly be effective. Thus the assertion that the problem of induction is intractable for REs has been couched in tentative terms.

[4] This illustration is modified from the RE entitled, “Monitoring Works. Getting Teachers to Come to School” (Hanna, 2005)