Project evaluation

Evaluation was embedded throughout the project. The evaluation was led by Joanne Kenworthy, Senior Lecturer in Linguistics and was carried out by the Cass School of Education at the University of East London (UEL).

Findings from the evaluation have directly feed into the content of this website. UEL has produced a user-friendly report of the evaluation of Listening to Learn, in addition to submitting contributions to academic and arts journals.

Key issues for evaluating similar projects

Issues of Ethics

In any evaluation the rights of participants need to be recognised; evaluators need to think critically about the processes of the evaluation task in relation to these rights.

Here are some questions that need to be addressed at all stages in an evaluation:

(Selected examples from Listening to Learn are provided.)

• What are the children/adults being asked to contribute?
• Might there risks or costs for participants - inconvenience/ time/sense of failure/ anxiety?
• Might there be benefits for participants—increased confidence or knowledge/ sense of satisfaction or achievement/ time to talk to an interested listener or peers/colleagues?
• How is privacy and confidentiality to be safe-guarded?

For example, In Listening to Learn, the children’s names were not used in any report; the project leader organised permissions for the use of any photographs; adult interviewees were assured of confidentiality and anonymity and were offered the chance to see the transcripts of their interviews.

• How are consent to participate dealt with and the participants’ right to withdraw respected?

Adult interviewees were given assurances of confidentiality and anonymity, and were told they could stop the interview at any time or decide not to answer a particular question.

• Has the purpose of the project been explained to participants?

When the work in the school began, the first sessions was use to explain to the children what the project was about, what they would be doing and when, and emphasise how important and valuable their ideas would be.

In addition to these questions, there are two issues that concern the stance of the evaluators themselves, and the uses of the outcomes of the evaluation:

• Throughout a project, evaluators need to think reflexively about their own expectations, level of involvement, and assumptions bout participants.
• Participants should be given access to the findings or outcomes of the evaluation.

There will be two reports, one for the children, and one for the partners in the project. These will be disseminated as widely as possible to interested people.

Designing an evaluation plan

• In order for evaluation to be embedded throughout a project, the general approach and working principles need to be considered early on in the planning and design or the project itself.
• Decisions need to be made about the skills needed by the evaluator(s) and their roles and responsibilities.
• Decisions about the work of the team need to include the relations and lines of communication with the project leaders/ management.

The evaluation team attended meetings with the Steering Committee to give a progress report three months into the project and to discuss drafts of the interim report. Discussions with the project leader took place weekly.

• Evaluation questions need to be closely aligned with the aims of the project. They also need to be clearly differentiated from the issues/questions that the project leaders might want to address.

In Listening to Learn the evaluators were not asked to assess the impacts on the artists of their involvement; for the project leader the experiences of the artists was highly relevant.

• Any evaluation plan designed to measure impact on participants must incorporate the collection of ‘baseline data’.

Before the regeneration partners had any contact with the children in Listening to Learn, they were asked if they had already had any previous experience of consulting with children and their families.

• If the plan involves a sample of participants to be tracked/ monitored, any criteria for sampling must be clear, and those who know the participants best should be involved in selecting the sample.

Listening to Learn’s work in the school aimed to be inclusive. Therefore, in the group of Year 4 and 5 children to be tracked, children with Special Educational Needs, different learning and behaviour styles, and particular home languages were included, with guidance from the class teachers.

• Choice of approach must be aligned with the evaluation questions and the overall structure of the project.

The innovative and exploratory nature of the project suggested a flexible ‘action-research’ approach allowing the evaluators to respond to any shifts in direction or unexpected issues that arose. During Stage Two there was a reorganisation of the peer-facilitation sessions. The evaluators needed to devise methods to capture the older children’s perceptions of these two different ways of working with their younger peers.

Carrying out the Evaluation

• All methods have disadvantages and advantages and evaluators need to constantly aware of these as they carry out data collection.

Semi- participant observation of the arts sessions in the school was used most frequently, as it meant that the evaluators would have periods of close contact with the children as they worked with the art forms. However, there were times when it was more appropriate and useful to be a full participant in an arts activity. For example, in one drama session an evaluator was asked by the children to act with them on their scene. Agreeing gave her access to the children’s development of their drama skills; refusing might have signalled lack of support and interest.

• The relationships that are established with participants are partly a product of methods chosen.

The decisions to observe as many arts sessions as possible meant that the evaluators came to be seen as familiar adults who were interested in everything that happened.

• Any specific activities designed to give insights into children’s responses need to be child-centred—they need to take account of the way children express their meanings.

Specific activities should:

Treat children as ‘experts’ in their own lives.

Offer children the opportunity to use a range of forms and media (spoken/written/visual) and enable children to choose from options.

Give children opportunities to be active. Games, excursions, guiding adults around their area, or (in Listening to Learn) using their newly acquired art skills will engage most children.

Use written text carefully. In designing questionnaires, evaluators need to consider literacy skills and incorporate alternative ways of responding (using an adult to read aloud and/or act as a scribe; using stickers to respond, etc.)

Give children as much control as possible. For example, children can interview each other instead of the adult interviewing the children.

Provide varied kinds of support that children can ‘talk to’, for example, photographs, maps, or drawings made by the children themselves provide a stimulus for discussion or question-answer sessions.

Use interviewing with care. Adults may feel ill-at-ease when being interviewed. In Listening to Learn video-taped interviews were designed where the interviewer was an imaginary character, and the children were in control of the camera which was set up in a private room with a special chair.

Use questions that have a clear audience, purpose or context that makes sense to children. An example is the question: ‘If there is another project like this one, what do you think children in a different primary school would enjoy/ not enjoy?’