Efficacy, effectiveness and scale

While Gray tends to use ‘efficacy’ and ‘effectiveness’ somewhat interchangeably in her report, it is worth digressing into a short consideration of these terms. Efficacy (as demonstrated in studies conducted under ideal experimental conditions) and effectiveness (as demonstrated in studies conducted under ‘real world’ conditions such as everyday clinical settings) exist in practice on a continuum (Singal et al., 2014). Many researchers consider the outcomes of effectiveness trials to be the most useful for clinical decision-making and have therefore argued for their routine inclusion in systematic reviews (Gartlehner et al., 2006). The relationship between practice-based evidence and evidence-based practice is reflexive and complementary (Barkham & Mellor-Clark, 2003). Well-designed observational and quasi-experimental studies enable everyday clinical practice and retrospective examination of records to also become part of the research picture and can similarly be included in reviews of quantitative studies (Maciejewski, 2020).

CAT practitioners may be understandably reluctant to engage in research that might prioritise experimental principles over the immediate and specific clinical needs of vulnerable populations, even though safeguards can and should be put into place (Nicholls et al., 2022). Those of us whose background studies are in arts or humanities may initially struggle with the language, let alone the procedures, involved in quantitative research, despite sustained efforts by leaders in the field of art therapy research to familiarise a focus on efficacy (e.g., Kapitan, 2012). Yet science/art and thinking/feeling are constructed and unhelpful binaries: systematic thinking and planning is important to the arts, and there is an inherent creativity in any good research design (Kapitan, 2018). When CAT practitioners do engage in a study of efficacy or effectiveness, it tends to be small in scale and of limited duration. Many pilot studies are promising but fail to secure the opportunity to exceed their stated limitations and build on their findings. Few of us have access to the funding, resources, time and expertise required to conduct a randomised controlled trial (RCT). This is where systematic reviews come into their own: by systematically selecting, ranking and analysing a group of primary research studies to produce convincing statistical evidence, including smaller studies with similar findings that by themselves would remain inconclusive (Gartlehner et al., 2006). Meta-analyses of systematic reviews maximise ‘estimated effect size’ (Paul & Barari, 2022), further compensating for limited scale.

Limitations

A statistical synthesis and discussion of overall findings, while impractical without more extensive time and resources, would have been a useful extension of the ANZACATA report. A further limitation of the report is that it does not discuss its own review methodology. We know that Gray has carefully conducted a search of CAT research studies with a significant quantitative dimension plus other reviews of these studies [4], then presented the findings in a table that enables readers to easily extract the information most relevant for their particular practice context. However, we remain unsure of the inclusion and exclusion criteria (McKenzie, et al., 2023), other than the obvious need to exclude purely qualitative studies when focusing on efficacy. We are thereby left to make our own assessment of ‘research quality’ [5] – which in simple terms refers to how the evidence has been produced and whether we can therefore rely on it – based on helpful but brief notations about each study’s research design. It is not clear why some, but not other, mixed-methods CAT studies have been included, and whether this and other apparent omissions are indirectly accounted for via the literature reviews included in the ANZACATA report. One of the more notable absences is the primary and secondary research into mental health recovery by Patricia Fenner, Theresa Van Lith and their colleagues (see Fenner et al., 2017; Van Lith, 2016; Van Lith et al., 2013), which may have been excluded from tabulated results because their work is already covered by the authors of other major reviews or because it did not meet unstated inclusion criteria. Some recovery-oriented research into CATs could have been missed by the report writer if the initial search process divided mental health into specific diagnoses. In the absence of an explanation of review methodology, such observations remain speculative.

In fairness, the strong emphasis on methodology or design that might animate academic interest in the context of a scholarly journal would be out of place in a professional report, but an endnote or appendix on review methodology could have strengthened the substantive claims for efficacy.

When limitations become likeable

The report focuses almost exclusively on the positive and beneficial outcomes of the CATs. This intentional skew simultaneously increases the publication’s utility as advocacy for our still underrated and under-resourced profession while limiting the report’s positioning in the conservative hierarchy of research. It is nevertheless to ANZACATA’s credit that, in commissioning this report, they did not reproduce the paradoxical effect of PACFA’s 2013 commissioned systematic review (Dunphy et al., 2013), which set such strict exclusion criteria that readers had no option but to think that the only worthwhile evidence that existed for the effectiveness of the creative arts therapies was conducted outside our region and/or was confined to music therapy. I have wondered whether, in that particular moment of PACFA’s project of professionalisation, their wish to appear as rigorous in evaluation of psychotherapy and counselling as our colleagues do in clinical psychology, while simultaneously unable to match the research funding psychology can attract, may have prevailed over the possibility of synthesising the sometimes anarchic range of quantitative, qualitative and mixed-methods CAT studies into something ‘considerable’ (see Van Lith et al., 2013). In contrast, this is a consideration that Gray’s review for ANZACATA strategically takes into account.

A limitation of this review of the ANZACATA report

A limitation of this review of the ANZACATA report is that the review writer is the Chief Editor of JoCAT, the association’s journal. ANZACATA fully supports the editorial independence of JoCAT and has not been involved in this review. However, it needs to be noted that the review writer’s structural positioning is not the same as that of an unaffiliated observer.

Research quality includes and is not limited to the quantitative

As Gray perceptively notes in her introduction, our affinity with affect may predispose us as CAT practitioners and researchers toward qualitative modes of enquiry. The ANZACATA report offers a corrective to a predominance of qualitative studies in the CAT literature and speaks in a language widely recognised by scientific communities.

This review of Gray’s report is therefore not the place to dwell at length on the well-established fact that qualitative studies can also be evaluated for rigour (e.g., Johnson et al., 2020; Lincoln & Guba, 1985; Mays & Pope, 1995). However, I do need to note the minor slippage into suggesting otherwise, through the repetition of the word ‘rigorous’ in the rationale for the parameters of this particular report, together with the reduction of the spectrum of qualitative approaches to the phenomenological in the well-intentioned moment of “honouring the value of human experiencing” (Gray, 2022, p.1). That phrase also gestures toward something else that has been strangely marginalised in the CAT literature, namely the voices and expertise of therapy and research participants (Balatti & Fenner, 2014), extending to a major under-representation of culturally diverse and Indigenous perspectives (Fenner, 2021), with recent work more radically advocating for coproducing research with participants (Springham & Xenophontes, 2021). Of course, the ANZACATA report neither claims to nor effectively negates the value of these other approaches to and forms of evidence.

Conclusion

A story of progress and potential

A great deal of research has been conducted in the ten and a half years since the PACFA review (Dunphy et al., 2013) was launched, as demonstrated by the selected articles tabulated in Gray’s report, many of them recent and including several major clinical trials as well as systematic reviews of the literature in their fields. While the report could have been clearer about its own methodology, the presumed need for reasonably strong exclusion criteria speaks as much to the pleasing proliferation of CAT research as to any questions about its quality. Gray has been given licence to bring diverse findings together into a coherent picture of what can be legitimately stated to others as scientific support for our practice. Rather than implying that the ANZACATA report should have included more confounding or more varied evidence, I suggest that we need both this report and more places, elsewhere in the scholarly literature, where we can conduct frank and fearless assessment of both the strengths and the shortfalls in our practice of both therapy and research.

Some of the most robust and convincing evidence for the CATs continues to come from mixed-methods research (Van Lith, 2017), including studies conducted in and beyond the ANZACATA region (e.g., Bowen et al., 2021; Griffin et al., 2023; Huang et al., 2021; McDonald & Holttum, 2020; Palmer et al., 2017), some of which are represented in this report. The triangulation of statistical with lived-experience data and, importantly, the reflexive integration of quantitative and qualitative analyses (Bazeley, 2022) can unite to amplify what ‘counts’.

The creative arts therapies inhabit a contemporary research landscape where science and the arts are complementary (Kaplan, 1999): where we can undertake quantitative or qualitative studies, or an integration of these methods, without diminishing our positioning in the hierarchy of professions. Within this landscape, The proven efficacy of creative arts therapies: What the literature tells us is a significant point of orientation for the CAT profession. I commend it to the ever-expanding readership of JoCAT[6]