Section 28 Clarifying how to ask QuIP questions in contexts when the final outcomes are not necessarily “changes”

28.1 Classic QuIP: the focus on changes

The QuIP protocol usually begins by asking respondents questions about changes in pre-defined domains of interest like nutrition within a reference period. Sources mention items important for them in that domain and work backwards from each to provide chains of antecedent causes. QuIP is mostly very interested in the extent to which respondents spontaneously mention an intervention of interest amongst the causal antecedents -– though it can be used for other purposes too.

Here, I’ll look at how QuIP doesn’t need to rely too heavily on the idea that the items we code have to be changes.

28.2 Even final outcomes don’t have to be changes

The way the very questions are formulated in QuIP means that, in QuIP, final outcomes (those at the end of an individual respondent’s causal chains) have to be expressed as changes (because that is what we ask people about). But that doesn’t have to be the case. In some contexts it might be important to ask about other “Differences made” by a project.

28.2.1 One-off events

The commissioner of this evaluation might have been involved in, say, providing the preparation of employers and is very interested in improving the retention of first-time workers within the reference period. But the individual students interviewed as part of this evaluation are not, from their own individual perspectives, describing changes of any kind; they are describing an experience which is by definition a one-off. They are describing Differences made, contrasts with hypothetical alternatives, not with prior states experienced by the respondents themselves. “I really knew what to expect and the people knew what support to give, much better than what my cousin said it is like where she lives.”

Weddings, births, transition to first employment etc, and their specific characteristics are all stand-out examples of events which are obviously important in individual life cycles and are often mentioned in QuIP interviews but which, at least from the perspective of individuals, can only with difficulty be characterised as changes within a reference period.

It’s an ethical fallacy (common in evaluation circles) that desirable changes have to have a significant trace arbitrarily far into the future. Good things can simply peter out. It is important that for example refugees are treated with dignity during a project, whether or not that somehow has consequences at the endline of a project. So the “final outcomes” of interest don’t have to be in the present. We can ask “how was your experience in the maternity ward, how did it compare with what you had expected, with what other mothers had told you?”

28.2.2 Maintaining the status quo

QuIP can also contribute to the evaluation of projects which aim to maintain the status quo, where the aim is for things not to change.

28.2.3 Status quo as a divergent counterexample

As I understand it, even when a positive change is expected and reported by many respondents, if we insist on only asking for changes (as QuIP suggests), we suppresses the experience of others who did not experience a change. As they can only provide causal chains ending in a change, their experience is missed out. To be sure, as QuIP generally only includes respondents who have been exposed to the project, if a subset of respondents do not report a positive change, at least we can count them. But the respondents might have other information to give about other important non-project factors when the status quo was maintained which simply goes missing if we only ask about changes. What if they would have been able to report that they didn’t get good crops because of the presence of a pest, but that information never gets written down?

Embracing the idea of “Differences” rather than “changes” would allow QuIP to evaluate projects with the aim of maintaining the status quo, although obviously the questions would have to be formulated a little differently.

28.3 Preceding items don’t have to be changes either

As I understand it, there is less insistence in QuIP that the antecedent items which respondents report as causes of the final outcomes themselves have to be changes: they might for example be “states” or “events”.

28.4 How to focus on the relevant contrast

The trouble is that it is not always obvious what we are supposed to be contrasting with the present state of things – isn’t this subjective?

Even insisting on “changes” – in the sense of differences from a baseline – does not remove this subjectivity, because what counts as the baseline is also up for grabs. If we ask a teacher how the new teaching method is affecting student performance, they are unlikely to think of a baseline derived from student scores at beginning of the year, as children will improve anyway. They are more likely to contrast performance with similar cohorts at the same time of year.

So in a specific research context it may be necessary to ask in an open way which suggests a variety of different axes of contrast: “how was your experience in the maternity ward, how did it compare with what you had expected, with what other mothers had told you, with your previous birth (if you had one)?”

In the variable-based approach, the global question is of the form “what causes what around here, in domain X?” which is less susceptible to this kind of problem; but even so, the questions actually asked in the field need to be tailored not only (of course) to the context but in particular to the intervention and the kind of Differences it is likely to make.

28.5 Drivers

28.6 Focus on initial items (“drivers”)

I think that classic QuIP gives a special role to the notion of drivers as the beginning of a causal chain; the project of interest is, of course, by definition a driver, but other “beginnings” get special status too? I’m thinking of Fig 5 in BSDR (2017), which shows a table of outcomes attributed to each driver.

The word “driver” seems to suggest that the beginning of a chain is the wicked initiator, and the intervening items are just innocent bystanders transmitting an impulse. But most real chains aren’t like that are they?

In this example, unemployment might be a relevant and actually present causal factor even without government ineptitude. I’d have thought you’d want to see “unemployment” in Fig 5 even though it might never appear at the beginning of a chain? In other words, does it matter to QuIP that an item (apart from the project itself) is at the beginning of a causal chain, does that give it a special status?

So perhaps we can say:

The basic unit of “classic” QuIP coding is the causal chain.
The final item in the chain is a change experienced by the respondent in a given domain during a given reference period.
Going backwards from the final outcome, each link in the chain codes how the influencing item causally explains (or partly explains?) the consequence item.
The logical nature of the preceding items is not specifically defined; anything is allowed which could count as a (partial) causal explanation of the item it influences. This can include events, changes during the reference period, and contrasts with hypothetical alternatives.
Each item and each causal link has to be something which the respondent themselves has directly experienced/witnessed.
Each chain is tagged with an assessment of the extent to which the final item (the change) can be attributed to the evaluated project.