Celebrating Age | MSC SERA website

PRACTICE EXAMPLE - MOST SIGNIFICANT CHANGE

Evaluating a national arts & culture programme for older people: Celebrating Age

Introduction to the programme and our evaluation

Celebrating Age is a £3 million programme of funding run jointly by Arts Council England and the Baring Foundation. A total of 33 projects have received funding in Rounds 1 and 2 of the programme, with varying start and finish dates between March 2017 and September 2021. The programme aims to support arts and cultural organisations, working in partnership with others, to take arts and culture to places where older people will find it easier to engage and to remove barriers to older people being included within mainstream spaces.

Imogen Blood & Associates, working in partnership with Mark Robinson of Thinking Practice and Lorna Easterbrook, were commissioned to conduct an independent national evaluation of the programme over both rounds of funding.

Why?

We decided to use MSC – alongside more traditional monitoring methods – right from the outset, because:

We could see a good fit with the ethos of MSC, and the participatory and creative aims of the programme;

Practically, it seemed to be the best way of drawing learning from 33 very diverse projects with different start and end dates over a four-year period and with insufficient budget to conduct primary qualitative data collection at each;

We were aware that some projects were already being evaluated by local evaluators; we hoped MSC could be compatible with these and draw national learning from them without too much additional work;

It can be very difficult for arts projects to gather evidence of their impact using traditional indicators, especially where they are working with older people (whose health and wellbeing may deteriorate for reasons which are beyond the control of the project), and where their ‘interventions’ are relatively small in terms of duration or frequency. We were aware that many arts organisations reported ‘anecdotal evidence’ of positive change to individuals, groups, communities and organisations, and we hoped that MSC would provide them with a tool to capture and utilise this evidence more systematically. With support from us and the creation of a community of practice between funded projects, we saw the national evaluation as an opportunity to build self-evaluative capacity, whilst also capturing evidence of impact and reflective learning.

How?

Projects came together at biannual Celebrating Age seminar events throughout the duration of the project (online from 2000, given social distancing). We introduced the concept of MSC to the Round 1 at the first of these events in May 2017.

We provided projects with a range of resources to support their use of MSC, including:

How-to-guides and templates to support story collection and reflection panels;
Phone calls to individual projects to help them apply the method in their own contexts;
Sessions at Celebrating Age learning events to share experiences.

We asked projects, in advance of the Celebrating Age event in September 2019, to:

Collect stories from participants (and/or other stakeholders);
Meet as a group to discuss the stories and identify the learning from them;
Share the story or stories the group selected as being the most significant and the rationale for selecting them in a short report template.

20 projects shared stories with us; attendees at the September event (which included Celebrating Age projects, people from the wider sector, and funders) discussed a total of 16 stories (people were divided into small groups, each discussing a collection of 4 stories). The groups summarised their reflections on what they felt was most significant about the stories they had discussed during the plenary session.

We had hoped to run a smaller national panel to review the stories selected by the participants at the September event in Spring 2020, but Coronavirus prevented this.

During 2020, we asked projects to gather stories from artists, professionals, host and partner organisations and reflect on changes to organisational and individual practice as a result of Celebrating Age. We suggested that this might be a good opportunity to ‘turn the table’ and engage older people to find out which practice and organisational changes are most significant to them. Projects were at different stages of doing this when Coronavirus struck. We plan to hold another national panel discussion (probably online) in Spring 2021 to reflect on these stories of organisational change, and on the impact of the pandemic. The findings of this will inform our final report on the programme in September 2021.

What was learnt about MSC that others could benefit from

Some projects picked up the concept and ran with it; others took longer to implement the methodology; some never did – sometimes because they already had well-established evaluation methods in place; sometimes because they saw it as ‘just another thing to have to do’.
Some projects really liked the holistic, collaborative and flexible nature of MSC – it worked particularly well in those projects which had the opportunity to build longer term relationships with a relatively small number of older people.
One of the significant advantages of MSC is that the structure, format and approach can be used flexibly and creatively to fit the style and structure of projects that are very different from each other.

However, some projects reported barriers and limitations, though some found ways to overcome them. For example:

It can be hard where people don’t have much to say;
It can feel quite formal, projects didn’t want to put people under pressure to identify ‘significant change’ and attribute it to the project; some people resist this, particularly if their engagement with the project is relatively short-term
The approach can feel intrusive: one project described how one care home resident had become deeply engaged during a dance session; she had raised her arms almost in a trance. The artist was able to observe and later relay this powerful image, but it felt too personal to ask her to recount it verbally at the end of the session.
It can be difficult where people have significant memory loss – people may not remember what you did in the session – they may experience feelings of joy (and sometimes this is evident!) but not be able to articulate that.
It can be hard to both collect monitoring data and stories of change from the same people.
Projects typically assumed that stories would be collected from older participants and discussed by paid professionals; some immediately saw the value of reversing this, i.e. collecting stories from paid professionals and inviting older participants to discuss them; others found this idea harder to grasp or implement.

The national leads from Arts Council England have seen the value in the approach: recognising that stories can be at least as powerful as numbers – if not more so – when seeking to influence policy.