Evaluating Impact

How do we know NUSTEM making a difference?

Five Year Impact Study Findings

When NUSTEM starting working with primary and secondary schools in 2014, we developed a five-year impact plan. This was intended to measure the impact of our work, as well as looking at children’s and young peoples’ attitudes to science and careers over five years.

Our first data collection took place in 2015 in a selection of our primary and secondary schools. Follow-up data collections were done in 2017 and 2019.

You can find our methodology and detailed findings here:

Primary Schools 5-year findings

Secondary Schools 5-year findings

Progression Tracking via NPD and HESA data

Summary of Primary School findings:

In primary schools, we looked at children’s career aspirations and knowledge, their self-concept and their concept of scientists.

It is possible to broaden the range of STEM careers children would consider and reduce the gendered nature of those career choices, but changing aspirations takes time. After 5 years we have made some changes to aspirations for some physical science careers such as engineer, but less impact on health science careers, and there is still a gap between the aspirations of different genders.
It is possible to reduce stereotypical views of scientists, shown by changing perceptions of scientists among girls. Again it takes time to make these changes. Changing perceptions of scientists is important as the more closely these perceptions match the girls’ descriptions of themselves (self-identity as kind and hardworking) there is an increased likelihood that they can see themselves as a scientist in the future.More research is needed to explore why perceptions of scientists did not change amongst the boys in the study.
We found it is more difficult to improve enjoyment and self-efficacy in science in the current educational climate, and we didn’t see expected changes in this area. That children are not confident in their science abilities is a concern to us. We would like to explore how this compares to confidence in other subject areas.
And lastly, it is difficult to disentangle the impact of NUSTEM’s work from the impact of many other influences on children in schools (teachers, school policy changes, curriculum policy changes, other school programmes/interventions etc). Having a Theory of Change which describes the intended change is important in making sense of these results, along with the flexibility to adapt if we find evidence that an intervention is not working as intended.

Summary of Secondary School findings:

In secondary schools, we monitored the changes in ‘aspects of science capital’ in year 7, year 9 and year 11 cohorts.

Changing perceptions of scientists: By 2019 fewer students were reporting that ‘people who use science in their jobs are very clever’. This is positive because it could potentially reduce the barrier to students entering science careers.
Changes in Parental attitudes to science: In 2019 more young people reported that their parents ‘would be happy if I became a scientist when I am older’, but fewer reported that their parents ‘have told me science is useful for my future’.
Confidence in Science abilities: We saw a drop in young people’s confidence in their science abilities between 2015 and 2019, and a widening of the confidence gap between boys and girls.
Enjoyment and value of science: There has been a decline in the number of young people who report that they enjoy science. Similarly, there has also been a decrease in young people’s perception of the value of science to themselves and to society. For both of these, there has been a larger drop for boys than girls.

Whilst the first two of these findings are generally positive, the reduction in the confidence in young people’s science abilities, their enjoyment of science, and their perception of the value of science are concerning. Our research does not allow us to identify directly what has influenced these changes, although a significant change to the content and level of difficulty of the science curriculum introduced in 2015 may have affected students’ attitudes.

Theory of Change linked Evaluation

Since 2019, NUSTEM have moved from one centrally-funded project and over-arching longitudinal research and evaluation scheme, to a number of different projects and interventions supported by different funders, each with their own evaluation requirements.

Central to all however is the NUSTEM Theory of Change. We ensure all of our projects and interventions have aims and outcomes linked to at least one causal pathway within the Theory of Change, and have developed a body of research instruments suitable for evaluating key outcomes within the Theory of Change among target audiences.

In this video, Julia describes the impact that working with NUSTEM is having on her school.

Evaluation Strategies and Models

We have also developed different strategies, models and methods for evaluating short, medium and long-term projects and interventions for NUSTEM.

	Short Term	Medium Term	Long Term
Aims	Aimed at increasing staff knowledge and understanding, and setting future direction	Aimed at finding out the success of individual programmes of activity and which ‘work’	Aimed at evaluating the overall success of sustained project’s in achieving long-term aims
Timeline	Evaluation takes place at the end of one-off events	Pre, post and (possibly follow up) evaluation of short courses and medium-term interventions	Baseline measures and repeat data annually, or bi-annually
Methods for use with children and young people	Feedback postcards	Pre and post questionnaires Worksheets that document learning journey (completed at beginning and end of project) Children’s work e.g posters, poems, art works etc	Baseline and annual or bi-annual questionnaires or data collection using research tools Annual or bi-annual focus groups Data-mining of datasets of pupil attainment, achievement and participation
Methods with teachers	Informal feedback from teachers and participants	Feedback through teacher forums Short reflection questions post project	Baseline and annual or bi-annial questionnaires or extended interviewing Focus groups to explore emerging themes
Methods with parents/carers	User-friendly feedback postcards at the end of one-off events	Pre and post questionnaires Regular feedback postcards	Baseline and annual or bi-annual questionnaires or data collection using research tools Regular feedback postcards Focus groups to explore emerging themes