Multi-agency services: monitoring and evaluation

A successful monitoring and evaluation strategy will help you to develop your own evidence base, assess whether you are delivering effective interventions, and modify them accordingly. This will assist you in communicating the value of your service to local partners, including schools, parents and other agencies.

Monitoring

A good monitoring system focuses on the information that your service needs to know to operate most effectively.

In short-term monitoring, the aim is to find out in a simple and efficient manner the reasons for an issue; for example, to find out which parent support groups have the highest attendance.

Long-term monitoring is useful for investigating trends over time, such as patterns of referral from different schools.

The key issues in any monitoring system are:

• Identify what information you really need and focus on collecting and recording this effectively.
  
  
• Gather this information from the start so you have baseline data.
  
  
• Gather it in the same format (where possible) over time so you can make comparisons and evaluate.

Your assessment, planning and review system will provide the core information about individual service users. If structured appropriately, it will collect information on:

• The needs of children and young people using your service
  
  
• The outcomes for those children and young people after a period of support
  
  
• Their views on the service they have received
  
  
• The views of their parents/carers
  
  
• Views of other agencies working with the team

In addition there are a number of other sources of information which may provide helpful monitoring information, particularly in relation to any whole-school or systems-based work in which your are involved, for example:

• Views of practitioners working with or within the service, including universal health and education staff
  
  
• Audits of the whole school or whole setting, for example in relation to behaviour and attendance
  
  
• The self-evaluation report (Form S4) that schools complete as part of the Ofsted inspection process

Evaluation

One of the key uses of monitoring data is to provide information for the evaluation of your service. Evaluations have two main related purposes:

• To answer questions about the impact of services and projects, including:
  - specific outcomes for service users
  - impacts for different groups of people or organisations
  - how and why services achieve or do not achieve their objectives
  
  
• To answer questions about the processes, structures and outputs in delivery and implementation of services and projects, for example:
  - what is delivered, to whom, and how?
  - what are the characteristics of participants?
  - who is not being reached by the service or project?
  - what are the barriers and challenges in delivery?
  - what are best practice solutions and lessons learned?

Evaluation is therefore important:

• As a vehicle for gathering accurate baseline data and contextual information about the service
  
  
• To inform ongoing decision-making and assess whether a service has achieved, or is on target towards, its aims and objectives
  
  
• To show what has happened as a result of the service being in place, including unexpected outcomes
  
  
• To improve practice and inform future strategy and planning
  
  
• To secure support - financial and/or in kind - for the service
  
  
• To meet any inspection requirements

Multi-agency services involve many different types of input, which means it can be difficult to isolate which aspects of the service are responsible for which outcomes.

For example, is the improvement in a pupil's exam results attributable to the adult learning programme her parents attended, which helped them engage more with her learning? Or is it attributable to the work her teachers did with her on problem-solving? Or the fact that she attended a nurture group which helped her self-esteem? In reality, it is likely to be a combination of all of these things, which has some practical implications for evaluating multi-agency services.

The team evaluating the extended schools pilot suggest that two things are important in evaluating complex multi-agency initiatives:

• First, services need access to sensitive measures of outcomes wherever possible. Some outcomes are difficult to measure numerically, but all can be assessed in some way. It therefore makes sense for services to develop a bank of indicators which they can use to assess the effects they are having.
  
  
• Second, services need to make their rationale or purpose explicit; identify the actions they intend to take in line with their service rationale; and track the extent to which their actions do indeed have the impacts they intend - ie helping the service meet its purpose and aims.

This suggests that the evaluation process needs to start when the planning of the service starts - to ensure that it is built around a well-informed needs assessment and a rationale for why particular actions will produce the sorts of changes that will meet those needs. It then needs to be supported by the collection of detailed monitoring data so that meaningful and ongoing evaluation can take place.

Reading and resources 

Evidence and Evaluation
One of the extended schools 'know how' leaflets, this short guide looks at why schools need to evaluate their extended activities and offers some tips around evaluation.

Evaluation of the Extended Schools Pathfinder Projects
See chapter four for information on the 'theory of change' model of evaluation described above.

An Introduction to Qualitative Research
Written by John Schostak of Manchester Metropolitan University, this guide covers the differences between qualitative and quantitative research, action research and guidance on how to carry out an evaluation.

Monitoring Whole School Practice to Promote Positive Behaviour and Attendance
Produced for the Secondary National Strategy, this guide provides an insight into school monitoring systems and methods for collecting information on some of the key issues that multi-agency services will also be interested in.

Extended School Self-evaluation Toolkit (ContinYou)
ContinYou has developed an audit toolkit called 'How are we doing? A self-evaluation toolkit for extended schools'. It is designed to encourage the involvement of headteachers, governors, parents, students, partner agencies, etc. Alongside the audit toolkit, an action plan has been developed to enable each school to move to the next point on the continuum. ContinYou now offers a consultancy service to support schools to use the toolkit. It offers schools a structured way to collect into a portfolio all their good practice and evidence of their progress towards being fully extended. The portfolio also works alongside the Ofsted self-assessment framework.