Having done the GP comparison report, I thought I’d also have a look at Understanding whānau-centred approaches: Analysis of Phase One Whānau Ora research and monitoring results. Te Ururoa and Winston have been at loggerheads over what we’ve actually learnt from this report. It says nothing about where the money has gone. But what it does attempt to say is:

What is the Whānau Ora approach? Is it happening? Is it working?

These are far more interesting questions. This is much larger report that attempts to take an action research approach – researchers alongside Whānau Ora Collectives as they developed and implemented their model of delivery. It’s a long report which is all the usual fluff for the first 16 pages before we get to the substance.

The Data

The data they collected was quarterly reports on “strengths-based and culturally grounded
measures that were based on whānau aspirations” and whānau satisfaction surveys between 2011-14. There was a quantitative analysis of over 200 monitoring reports from 34 collectives and 895 whānau surveys and a synthesis of the qualitative data in the monitoring reports and the 60 action research reports. The initial data is interesting: over three years 34 collectives should have provided over 300 reports; and 895 whānau satisfaction surveys is about nine whānau per year for each collective, which is poor response rate; I do wonder about the reliability of the sample given 5,499 whānau are working with navigators and 6,933 plans have been developed at June 2014. It can be very hard to get workers and whānau focused on actually doing surveys.

Te Puni Kōkiri researchers identified the data limitations for themselves:

• the research was confined by the documentation with gaps in information related to funding, contracting and policies at a national level. There is no comparison of the different models used between collectives;
• selection bias in surveys as providers could select which whānau participated;
• Whilst the research could find improvements, they didn’t have the data to say exactly what the improvement was. Strangely here they noted that they didn’t have whānau-level data; so when the report claims the measures are based on whānau aspirations, what do they actually mean?;
• Providers were clearly at a variety of stage of readiness for Whānau Ora; in other words there were some doing Whānau Ora, there were some just talking about doing Whānau Ora. Also some the collectives clearly weren’t working together very well, and were a bit cautious about the researchers finding out what was really going on.

The Findings

Whānau engagement

Most whānau who engaged ended up doing a whānau plan, but only 58 percent had engaged with a navigator. So 42 percent whānau are at a service that is whānau-centred, but not using whānau navigators. There’s not enough navigators nationally. That’s followed by this insight: “income and employment, often an important determinant of economic gains, appeared to be less often addressed [in plans] than those services addressing health, education and social needs.” I’m probably as unsurprised as the researchers. Most Whānau Ora Collectives are dominated by health providers, so they deal with health and soclal needs because that’s what they know. Indeed, housing, employment and finances only made up one in five of the goals set. On average whānau with plans were achieving 68 percent of their goals. I’d be happy with that, though unfortunately we have no idea what sort of goals are being set.

Whānau improvements

This section worryingly starts with an assertion that because of the variety of outcomes and models it’s too complex to provide any measures. That’s actually probably true, but then there’s a bit of a bold statement that “results indicate that higher-level and extensive outcomes were achieved by whānau even though they were not collected across the board.” Yeah. You can’t really say that if the data wasn’t collected to prove it. So what they have are the satisfaction surveys in which whānau said there had been “big” improvements. I really want to believe that, but this is a small sample with a high risk of selection bias that provided no direction to those surveyed as to how to measure “big”. This section’s smoke and mirrors.

Service delivery and “progressive benefits”

These next two section frustrated me. The researchers are trying to apply a statistical relationship to the “average rating scores around service delivery… correlated with average improvement scores around intermediary and higher-level outcomes.” You can’t really do that with any great certainty as the relationship between those two things and what those two things actually mean to the person answering the survey is unclear at best; so whilst there appears to be some improvements, overall they’ve correctly noted a “weak correlation between service delivery and higher-level outcomes” and blamed that on a lack of time. I personally don’t think the data they’ve gathered lends itself to this type of analysis.

Service delivery and improvements

But they carried on with the statistical correlations into this next section, with the same results: looks pretty good, can’t be absolutely sure. This is a bad use of statistical correlation, because the data is qualitative and it has been grouped thematically (the fancy term they’ve used in principle components analysis) so what they end up with are some silly correlations: such as a correlation between knowing your whakapapa and smoking cessation. Someone needed to take a breath and question what quality this form of analysis was providing.

Whānau centred approaches

This narrative approach is better as their data sources work well with this form of analysis. And the findings are useful. Across the collectives, they’ve identified what is a Whānau Ora approach:

• effective relationships
• whānau self-management
• capable and skilled workforce
• whānau needs and aspirations are at the centre of service development
• a supportive funding environment
• everything is culturally anchored practice in te ao Māori
• whanaungatanga is the tool to connect and build whānau capability.

Great. This is exactly what we’re after. Now we need to know from this report if that is happening. Well, that’s not so clear. The researchers pad out their findings by saying “in general” collectives and providers are doing this. They then say that providers commonly said “it is about whānau” and four out of five said one size does not fit all which I suppose could mean they are flexible and whānau centred. Four out of five providers highlighted meaningfully engagement with whānau as important and that whanaungatanga is important. Collectives identified that whānau having self-autonomy and a sense of identity are key to better outcomes. Pretty much every collective could identify that they were training their workforce and they were culturally competent (which I’m relieved about as most are Māori and Pacific providers). The barriers to the Whānau Ora approach were whānau in crisis, providers in silos and the same old competitive funding environment and a misunderstanding of the Whānau Ora approach.

The most important finding for me is that four out of five providers tried to show flexibility for whānau needs and to re-orient themselve to meet whānau needs. which is great but not the full expression of this model above. I was worred to see that leadership is far more problematic: most collectives were demonstrating leadership in their wider networks, they didn’t feel government agencies were, and were only fifty fifty on iwi providing leadership. Only one in four providers had made any changes in their contracting arrangements to meet whānau needs.

So do we know what a Whānau Ora approach is? Yes.

Is the Whānau Ora approach actually happening? Not consistently.

Is the Whānau Ora approach working? There are positive improvements, but it’s a bit hard to tell for certain.

Overall then the researchers end up using the same word used by so many of us when talking or writing about Whānau Ora: potential. Whānau Ora has the potential; it hasn’t achieved its potential yet. I genuinely hope the Minister focuses on the negatives and demands the changes this report identifies need to happen. The approach taken today of waving the report and saying how great it is, is doing whānau a disservice. Putting a little acid on the funders and collectives to get it right is going to turn a good idea into a great service.