An evaluation puzzle: “Many hands”
In its report, CARE say they saved the villagers from being swept away in the flood.
A critic argues, even if CARE hadn’t intervened (counterfactual), the villagers wouldn’t have been swept away, because there was a surplus of aid agencies around with suitable boats, and most of them in fact had nothing to do. Ergo, CARE’s contribution didn’t make any difference.
You are asked to comment.
An evaluation puzzle: Loaves and fishes
Oxfam fixes the road after the flood, and Save the Children fixes the school.
The school couldn’t have been fixed without the road.
Both of the charities report to the UN’s standardised reporting format, and both record their contribution as “one school fixed”. Both argue: if our organisation hadn’t stepped in, the school wouldn’t have been built (counterfactual) but we did step in, and it was. Ergo, our organisation caused the fixing of the school.
So when the UN aggregates the reports, will it find that a total of two schools were repaired?
You are asked to comment on the reports.
An evaluation puzzle: “Freak weather”
Two NGOs prepare ships to help people stranded after a tsunami on a remote island.
NGO A is well-prepared and sets sail in time.
NGO B has barely enough equipment and leaves too late to do any good.
A freak hurricane sends ship A back to base but slings ship B miles ahead.
Ship B reaches the island and just manages to save many lives.
Ship A still manages to arrive on time but its help is no longer needed.
Which NGO was most effective? Which had the most impact? Which should be praised the most? Did NGO A fail?
An evaluation puzzle: “Billionaire”
A billionaire left 10 million EUR in his will to establish a trust, with instructions that it should be used to “just do good”.
During the ten years since then, support for same-sex marriage has shifted from 10% to 80% public approval, and almost all liberals are now in favour.
In the ninth year, the trust gives 1 million to a campaign for a law on marriage equality, which substantially contributes to the passing of a law in the tenth year.
We don’t know for sure, but most likely the billionaire, like most of his peers and friends, did not support marriage equality when he was alive. But most of his peers and friends now support it.
Now, in the 11th year, you are asked to evaluate whether the trust was run according to the wishes of the billionaire.
A starter kit for reproducible research with R
This accompanies my draft short article for the UKES Bulletin: A reproducible workflow for evaluation reports.
Here are just two files which together get you started with a bare-bones reproducible research project.
- Install and install R and RStudio.
- Download this little example of a reproducible source file and an accompanying example Excel file describing the heights and ages of a bunch of boys and girls in a school.
- Save them together in a new folder.
- Double-click the source file to open it in RStudio. The source file has a couple of commands to read the Excel file, clean it and make a table and a chart, as well as plain text which becomes the headings and body of the document.
- Press the blue “Knit” button in RStudio, you will get a beautiful Word document.
- Try editing the text and then press “Knit” again to see what happens.
How do you explain reproducible research to clients?
Most of the statistics work I do now is reproducible research - this
can offer a big advantage for clients but of course that doesn’t
necessarily mean they realise it …
Below is a text I have been working on to explain
reproducible research. Would be very interested if anyone has any
better ideas ….
Short version
Reproducible research: wherever appropriate, all tables, graphics and statistical analyses in my reports are produced from a “source file” which contains the usual headings, text etc., as well as machine-readable code for producing each table and graphic. Then each time I save a new version of the report, a statistics program called “R” loads the raw data, cleans it, does calculations, replaces the code with the corresponding tables and graphics, and produces a Word, PDF or web document. That means consistency in presentation, up-to-date results, less chance of error, and complete transparency because it is easy to independently check the results.
Long version
Wherever a report includes quantitative data, I produce it using a “reproducible” workflow: graphs, tables, and even calculated data such as totals in the middle of text passages are produced automatically (according to instructions in a “source file” using a statistics program called “R”) each time I save a new version of the report.
Advantages for the client:
- Elimination of errors because there is no manual cutting-and-pasting or editing of data in the original data files, or manual editing of tables or graphics: the original data files (e.g. responses to questionnaires) are not touched at all but are loaded from scratch each time.
- Faster workflow because I can produce preliminary and pre-final report versions even before all data collection is completed.
- Better presentation because it is easy to make global changes to the way data is presented (e.g. to change the font size on dozens of graphics with one click).
- Freshness of external data because the latest versions of any data from other sources, e.g. World Bank data, will be loaded live from online repositories to ensure “freshness”.
- Transparency & verifiable accuracy the client or others can, if desired, use the “source file” to repeat these calculations, see exactly how they are arrived at, and verify for errors.
- Openness and extensibility because the client can extend the analyses in later months or years — or hand them over to others if desired — without having to start from scratch.
Things for the client to bear in mind:
- My usual workflow is to produce preliminary versions of the report(s) online at my own domain so you can follow progress. This will update automatically so you will always see the latest version. I can produce Word and PDF documents too, but if you want to make preliminary comments it is better to use email / Slack / Skype etc rather than doing it inside those documents.
- I will usually produce the final draft in Word format. Up to that point, if you say, e.g. “I want all the table headings up to page 22 in a larger font” or “I want to exclude the data from before 2011 from the whole report”, I can make such changes very easily. After that, it is no longer so easy to make global changes to the appearance and content.