For context and further information this column is followed by an opinion expressed by Professor of Philosophy Tim Dare at the time when National Minister Anne Tolley pulled the pin on proceeding with similar work in New Zealand.
The New Zealand Economics Association annual meetings are a great way of keeping abreast of what the country’s economists are working on. And sometimes they’re downright depressing. At last week’s meetings, Auckland University of Technology’s Professor Rhema Vaithianathan’s keynote explained what she’s been up to over the past decade. Her team has been helping American child protection services to do a better job protecting kids. Child protection work is grim. Officials balance two terrible kinds of errors. Over-zealousness means a lot of families will be put through a painful wringer unnecessarily. But under-intervention means some kids who could have been helped will wind up abused, hospitalised, or killed. Unless you can find a way of reducing both types of errors. And Prof Vaithianathan’s team found a good one. Child protection workers have a mountain of administrative data for making decisions on whether to intervene in response to a call, but only about ten minutes to make each decision – then on to the next case. It is impossible to regularly make good decisions faced with that much complexity and that little time. Prof Vaithianathan’s team reduced complexity by turning data into a predictive score laying out the risk each case posed, to help child protection workers make the right call. They started the U.S. work in Allegheny County, Pennsylvania, because Allegheny wanted to use data to make better decisions. The programme’s success inspired others to try it out. And a later randomised control trial showed that the system reduced child hospitalisation by a third. It also reduced the bias that case workers otherwise bring with them in making assessments. The risk scored meant more high-risk white families received help and fewer low-risk black families had to deal with child protection services. It’s a great story. The depressing part? The work started here in New Zealand. It was killed by Anne Tolley as Minister, who described it as experimenting on kids. And the subsequent Labour government showed even less interest in data-based approaches. One third fewer hospitalisations for children in risky families. But not here. In America instead. Thanks to Kiwi researchers, who were chased away from doing the work here. An innovative American county can try something new and let others follow. New Zealand’s centralisation means a single bad Ministerial decision can cause a lot of harm for a very long time. Far better policy, and outcomes, are possible. Even here. But voters have to demand it.
Eric Crampton is the Chief Economist at The New Zealand Initiative. This piece was published in the NZ Initiative’s newsletter, Insights, 7 July 2023
PROFESSOR TIM DARE: Anne Tolley's 'lab rats' call inflammatory political rhetoric
A minister sees a briefing paper with a proposal to test a computer model designed to identify children at risk of maltreatment. She reacts strongly.
"Not on my watch!" she writes in the margin, "these children are not lab-rats". The study is shelved.
The media obtain the briefing paper, complete with the marginalia, and publicise it.
The Opposition seize on the lab-rats cry and use it in the House against the Minister of Social Development and the ministry.
Should we feel relieved? Have we averted another unfortunate experiment? No.
The minister's reaction, and the media and Opposition response to it should make us feel uneasy.
The problem is not the shelving of the study – though that was a mistake too – rather it is the chilling effect of the knee-jerk and political response to an attempt to produce evidence for important social policy.
Science collided with politics, and politics won.
The study was an observational study intended to test the accuracy of a computer model designed to identify children at risk of maltreatment.
It would have seen all children born in 2015 rated for risk. Data, which would have been collected over the next two years in any event, would have been analysed to see if the computer model had accurately identified those who were in fact been maltreated over that period.
The study did not involve any new or untried interventions: that is what it means to say it was an observational study.
During the period of the study, the children being observed would have received exactly the same services they would have received if there had been no study.
In the ministry's words, the study "would not have displaced standard response. Agencies such as Child Youth and Family and the Police would have, at all times, continued to act on notifications in relation to vulnerable children, as they do now. All children would have received the full range of support available from agencies".
The study did not propose to get the children or their families to do anything they would not otherwise have done.
It did not involve the withdrawal of any services.
Any notification that a child was at risk – notifications generated just as they are now, and independently of the computer model being tested – were to have been responded to exactly as they are now.
The 'observations' were entirely in the background: producing an automated risk ranking from data already held, and looking through records of substantiated abuse at a future date.
The study posed no risk to the children: it did not involve any intervention.
Now, of course, observational studies can raise significant ethical issues.
It is probably impossible to obtain consent from everyone in a study of this size; we know that the cases of verified abuse – the ones we would have looked for in the computer predictions – will not include all cases of actual abuse, and we might be concerned that it's easier to hide abuse if your GP thinks you're a good, hard-working, decent couple than it is if you're a beneficiary.
Those are the sorts of issues that would have been discussed by an independent ethical review of the research.
Perhaps such a review would have found the problems insurmountable and refused to allow it to go ahead.
But none of this warrants the knee-jerk dismissal of the research in a 10-word comment in the margins of a briefing paper.
The suggestion that the study would treat children as lab-rats is inflammatory and misleading, but rhetoric won the day.
The failure of anyone to take the trouble to try to substantiate the minister's initial concerns, rather than latching on to misleading rhetoric, bodes ill for the possibility of anything like a genuine research-driven approach.
That is something we should all be worried about.
Child maltreatment is an appalling and recalcitrant problem and New Zealand's record is particularly poor.
We will make progress in these areas only if we seek good evidence-based policy.
We must not stumble along relying on experience and anecdote: it leads us astray.
We owe the kids a better, more informed, approach.
Tim Dare is a professor of philosophy at The University of Auckland. In 2012 he was contracted by the Ministry of Social Development to provide an ethical analysis of the predictive risk modelling tool this study was intended to test.