Section 27 Limitations to the proposition-based approach

27.1 Problems within the proposition-based approach

27.1.1 Problem with only coding personal observations rather than personal causal knowledge

Suppose a new road is built (this is not the intervention), and nearby farmers have worse yields. I think James says we can process the interview testimony of farmers living near the road (whose yields have decreased) as new road in neighbourhood -–> yields decreased but not the testimony of more distant farmers (whose yields did not decrease), even though they are in touch with their peers and have personally experienced the “other half” of the counterfactual, and can also attest to the above statement. (Yet I think James says it is OK to code this kind of information when the farmers are reporting together in a focus group.)

From a variable-based perspective, this might seem bizarre. But from the perspective of, say, a journalist, or an evaluator conducting an ordinary summative evaluation of a project, it is a completely normal way of thinking: “we don’t want people’s theories, we just want the facts”.

This is a massive strength and weakness of the propositional approach. The thing is, I’d say that what counts as a simple fact about causation actually rests anyway on people’s underlying theories. We certainly don’t want farmers to tell us merely “I noticed the training. And I noticed my crops were better. That’s all I can say”. That reduces QuIP to a kind of soft self-report questionnaire with a dangerously low N. No, we want them to say “I experienced the training, I learned a lot, I could do things in new ways which make sense to me given what I know about farming, and I wasn’t surprised to see my crops were better”.

Evidence based on testimony from respondents who have “actually experienced” all the parts of a causal link (yes, I got the new seeds, and yes, I got better harvests, and yes I think the one is the cause of the other) usually carries a lot more weight for the reader than hypothetical statements. On the other hand, I think this oversells the importance of the experience of co-occurring events (which opens QuIP to criticisms about small sample size), as if our respondents were just uninformed scientific observers of random events, and underplays the importance of respondents’ more fundamental, underlying knowledge about this kind of causal link which is not only derived from any particular chain of events.

27.1.2 Problems with establishing common codes using difference of degree

Classic QuIP does already take one very important step onwards from the most basic and literal coding of individual causal claims, namely by establishing common codes for similar items across sources. This is already a kind of aggregation. For example, if one source mentions “some health problems” and another mentions “severe health problems”.

In a variable-based approach this could be easily dealt with for example by saying that the effect of two different items from the two different sources have a different strengths; and the variable “health problems” can have different values or levels. But if “health problems” is just a proposition, the difference in degree gets lost; conservatively, “health problems” can only be interpreted as the weakest proposition which could fall under that title.

Fuzzy Sets Ragin and Pennings (2005) provide an interesting approach: the items (like “unemployment” or “democracy”) can carry on as monolithic ideals, but respondent claims about them can be moderated according to degree of accordance with that ideal, aka degree of membership of that fuzzy set. However I think this ends up being a kind of variable-based system.

27.2 Limitations: Problems with extending the proposition-based approach

27.2.1 No way to code multiple causation

In the proposition-based approach, there is only one influencing item for each consequence item in a causal chain.

(However, the same source may mention “the same” item as part of different chains – when mentioning several chains leading to the same final outcome, or when chains share an intermediate or initial item.)

Allowing causal packages with more than one influencing item is not as simple as it sounds, not least because of the provision “items have to have been individually experienced”.

In this case, from a variable-based perspective we can see that there is a simple, continuous outcome. The project has a small, plus effect on it, which is outweighed by another larger, minus influence. The overall change (the only thing which the classic QuIP protocol can record) is a moderate reduction in income.

There is no way even of saying “the drought only led to a small drop in income; if the project had not happened, the drought would have led to a larger drop in income” because in the proposition approach there is no way to express this gradation of income.

The logic of “attribution” coding is tricky here. On the surface, this is a negative change, a negative outcome, which is also (partially) attributable to the project! This would make the project seem bad, whereas in fact the specific contribution of the project is (from a variable-based perspective) positive.

Also the requirement that “the items and causal mechanisms have to have been actually witnessed by the respondent” is harder to understand/implement here. It’s not quite clear what this means in the case of two opposing, if independent, forces. If I watch two ropes pulling on an object with equal force in opposite directions, do I really witness the causal effects of the two opposing forces in the same way that I might witness just one rope pulling unopposed on the same object?

I can’t just directly witness how two forces interact to make something else happen. I can only use my existing expertise and/or tweak the mechanism to find out how it works and/or observe over time and/or observe my peers. And that means elaborating a theory, going beyond direct experience.

It is very difficult to extend the propositional approach to even these very simple, additive / subtractive cases.

27.2.2 Problems with expressing difference of degree

The cousin of the problem with coding similar outcomes with different degrees (see above) is that the propositional approach has no way to code statements where the effect of B on E is explicitly relativised, e.g. “my lack of skills maybe contributed to me losing my job, but it wasn’t that important”. This already implicitly introduces the idea of additional causes, see above.

27.2.3 Explicit causal knowledge

I think it is also important that what is to be coded is an explicitly causalclaim. It is not enough that the respondent says “I noted that X changed. I also noticed that Y changed.” The QuIP protocol is designed to draw on respondents’ underlying causal world-view, and not primarily on observation of co-occurrence. The variable-based approach says this explicitly: we want to get at people’s theories. I don’t know how the propositional approach can distinguish between “I noted that X changed. I also noticed that Y changed.” and “I noted that X changed. I also noticed that Y changed; X caused Y.”

27.2.4 Vulnerability to luck / coincidence

Like most practical evaluation techniques, QuIP is vulnerable to over-interpretation of happenstance. For example, if a perfectly-conceived and prepared project is denied success by an incredibly unlucky meteorite strike at the last minute, QuIP is unable even to begin an evaluation, because there are, unluckily, no “positive changes” to put at the end of causal chains. QuIP is not alone in this: most narrative approaches to evaluation would have the same problem. Whereas a broader variable-based perspective5 might ask to what extent the project was the right kind of tool to address the problem, regardless of the fact that it was denied success in an unforeseeable way at the last moment.

This is the difference between evaluating overall success potential (variable-based; most interesting for scientists and policy-makers) and actual success / failure (proposition-based; most interesting for journalists, law courts, and Ambassadors).

There is a big tension between plausibility and validity.

The Ambassador might well have trouble understanding why an evaluation of success potential might give a project full marks even though it was ultimately unsuccessful.

(The same problem works the other way round, when a poor project “achieves” positive changes via lucky pathways.)

27.2.5 Unable to code necessary/sufficient conditions

Many other causal connections which we might want to code are essentially someone’s more or less formal theory, going beyond the direct one-off connections which the propositional approach is so good at capturing. For example, you can’t directly observe that X is necessary for Y. That claim rests on pre-existing causal knowledge, or is the result of more or less formal experimentation. So classic QuIP can code the observation “water made my crops grow” but not “water is necessary for my crops to grow” because that goes beyond a mere observation.

27.3 Weaknesses of the variable-based approach

Events are easy to capture with a proposition-based approach and harder to capture with a variables-based approach. So the assassination of Arch-Duke Ferdinand was a causal trigger for WWI, but it is hard to conceive of that assassination as a variable, (something that would still be around but not actuated in an alternative history in which he never went to Sarajevo).

27.4 What is to be done? Suggestions for QuIP

  1. Just stick to propositional coding. It is simple and can code the majority of statements. We can aggregate items and links as now, e.g. “X was mentioned 20 times” but go no further in our interpretation of what that means. Accept that we won’t be able to directly code multiple causation, necessary conditions, links with different strengths, statements about lack of change, etc. This approach has nevertheless worked well so far. It is natural, relatively easy to apply and explain, produces highly plausible evidence, and has a very natural and persuasive way of presenting the extent to which a project seems to have led to key outcomes, which is what commissioners want most. In particular it produces “golden threads”: when one respondent explicitly mentions a causal chain from the project to an outcome.

  2. As James suggests, coding can be done with the propositional approach and then a variable-based theory built up from that. But this does not get round the limitations of the propositional approach when doing basic coding.

  3. Use a variable-based model, as already discussed, but to make more effort also to preserve the simplicity and plausibility of the classic model. For example, the existing method of attribution coding can put special emphasis on “gold threads” when and if they appear; plus, “gold threads” can to a large extent be identified automatically.

  1. One could argue how “theory-based evaluation” fits in here↩