Need for collective evaluation measures
It is important for evaluators to include collective as well as individual measures when considering what constitutes a good outcome, says Professor Mason Durie.
“We are better at devising indicators that reflect the status of the individual in society but we are not as good with indicators that reflect the collective outcomes, whether they are for families or communities or populations such as Maori, Pasifika peoples or Asians,” Mason, Assistant Vice-Chancellor (Maori) and Professor of Maori Research and Development at Massey University, told the Bulletin.
“A key issue in evaluation is to make sure when we are talking about outcomes whether we are employing the right measures.”
It was important not to measure the position of individuals in society and regard that as an adequate measure of society as a whole without also measuring groups in society. “Sometimes we just draw the individual statistics together and think we have got a picture.” He said there were some outcome indicators that were universal. “But there are other indicators which are population specific. If you are looking at best outcomes, you need to use a combination of universal and population-specific outcomes rather than expecting everyone to be accommodated in the universal. Research does not always do that now.”
Mason has been involved in research developing instruments to measure best outcomes in mental health for Maori. “When we were developing them, we heard that most people tended to measure mental health by the presence or absence of symptoms – for example, a good outcome would be if a person’s depression was gone, or they no longer had delusions.” But a Maori perspective of health includes spiritual, intellectual and emotional, physical and family connections – and a good outcome would have gains in those four areas, thus reflecting a Maori world view in the outcome measure. Most mental health outcomes measured only intellectual or emotional gains, and paid little attention to gains in spiritual or physical health. “Yet they are critical to the notion of good outcomes for Maori.”
Mason was under no illusion as to the difficulties involved in outcome measurement. “You have to wait a long time for outcomes, and there are a lot of variables that affect an outcome. So there are problems in getting outcome measures to measure the efficacy of a particular programme. But we can do better than we are doing. We tend to measure efficacy on volumes – the number of passes in NCEA, or the number of people treated in health. But outcome measures need to reflect not only gains in a particular sectoral sense but gains that reflect the aspirations people have for their own society.
“In education, for example, a student may go through the education process and come out with knowledge that will equip them to participate in society generally, but it may fail to equip them to participate in their own specific society, be it Maori, Pasific or Asian. In education, a process that does not recognise what would be needed for that person’s particular life, that simply assumed everyone was the same, falls short of reaching a good outcome.”
With respect to the ethics of best practice in researching ethnic groups, Mason said the same principles applied there as for any social research. “That is, you have people’s buy in and active participation, not their acquiescence.”
He said research must be linked to the world views, priorities and aspirations of ethnic groups, so research did not plunder but helped to build. “But that’s true of all good research in a community. You would expect to establish good links and be able to communicate and get buy in.” Good research involved people early in the design and it was important to provide feedback. “Don’t take the respondents for granted. You have an obligation to let them know the results of research and to show how they might benefit from the results directly or indirectly over time.”
Mason said in one sense the notions were ethnic specific but they also applied to all research. “A lot of Maori values and notions about research have helped articulate best practice for research. People say we shouldn’t only do that for Maori. We should do it for all research.”
