Evaluating Impact
How do we know NUSTEM making a difference?
How do we know NUSTEM making a difference?
When NUSTEM starting working with primary and secondary schools in 2014, we developed a five-year impact plan. This was intended to measure the impact of our work, as well as looking at children’s and young peoples’ attitudes to science and careers over five years.
Our first data collection took place in 2015 in a selection of our primary and secondary schools. Follow-up data collections were done in 2017 and 2019.
You can find our methodology and detailed findings here:
In primary schools, we looked at children’s career aspirations and knowledge, their self-concept and their concept of scientists.
In secondary schools, we monitored the changes in ‘aspects of science capital’ in year 7, year 9 and year 11 cohorts.
Whilst the first two of these findings are generally positive, the reduction in the confidence in young people’s science abilities, their enjoyment of science, and their perception of the value of science are concerning. Our research does not allow us to identify directly what has influenced these changes, although a significant change to the content and level of difficulty of the science curriculum introduced in 2015 may have affected students’ attitudes.
Since 2019, NUSTEM have moved from one centrally-funded project and over-arching longitudinal research and evaluation scheme, to a number of different projects and interventions supported by different funders, each with their own evaluation requirements.
Central to all however is the NUSTEM Theory of Change. We ensure all of our projects and interventions have aims and outcomes linked to at least one causal pathway within the Theory of Change, and have developed a body of research instruments suitable for evaluating key outcomes within the Theory of Change among target audiences.
We have also developed different strategies, models and methods for evaluating short, medium and long-term projects and interventions for NUSTEM.
Short Term | Medium Term | Long Term | |
---|---|---|---|
Aims | Aimed at increasing staff knowledge and understanding, and setting future direction | Aimed at finding out the success of individual programmes of activity and which ‘work’ | Aimed at evaluating the overall success of sustained project’s in achieving long-term aims |
Timeline | Evaluation takes place at the end of one-off events | Pre, post and (possibly follow up) evaluation of short courses and medium-term interventions | Baseline measures and repeat data annually, or bi-annually |
Methods for use with children and young people | Feedback postcards | Pre and post questionnaires
Worksheets that document learning journey (completed at beginning and end of project) Children’s work e.g posters, poems, art works etc | Baseline and annual or bi-annual questionnaires or data collection using research tools
Annual or bi-annual focus groups Data-mining of datasets of pupil attainment, achievement and participation |
Methods with teachers | Informal feedback from teachers and participants | Feedback through teacher forums
Short reflection questions post project | Baseline and annual or bi-annial questionnaires or extended interviewing
Focus groups to explore emerging themes |
Methods with parents/carers | User-friendly feedback postcards at the end of one-off events | Pre and post questionnaires
Regular feedback postcards | Baseline and annual or bi-annual questionnaires or data collection using research tools
Regular feedback postcards Focus groups to explore emerging themes |
We may request cookies to be set on your device. We use cookies to let us know when you visit our websites, how you interact with us, to enrich your user experience, and to customize your relationship with our website.
Click on the different category headings to find out more. You can also change some of your preferences. Note that blocking some types of cookies may impact your experience on our websites and the services we are able to offer.
These cookies are strictly necessary to provide you with services available through our website and to use some of its features.
Because these cookies are strictly necessary to deliver the website, refusing them will have impact how our site functions. You always can block or delete cookies by changing your browser settings and force blocking all cookies on this website. But this will always prompt you to accept/refuse cookies when revisiting our site.
We fully respect if you want to refuse cookies but to avoid asking you again and again kindly allow us to store a cookie for that. You are free to opt out any time or opt in for other cookies to get a better experience. If you refuse cookies we will remove all set cookies in our domain.
We provide you with a list of stored cookies on your computer in our domain so you can check what we stored. Due to security reasons we are not able to show or modify cookies from other domains. You can check these in your browser security settings.
These cookies collect information that is used either in aggregate form to help us understand how our website is being used or how effective our marketing campaigns are, or to help us customize our website and application for you in order to enhance your experience.
If you do not want that we track your visit to our site you can disable tracking in your browser here:
We also use different external services like Google Webfonts, Google Maps, and external Video providers. Since these providers may collect personal data like your IP address we allow you to block them here. Please be aware that this might heavily reduce the functionality and appearance of our site. Changes will take effect once you reload the page.
Google Webfont Settings:
Google Map Settings:
Google reCaptcha Settings:
Vimeo and Youtube video embeds:
The following cookies are also needed - You can choose if you want to allow them:
You can read about our cookies and privacy settings in detail on our Privacy Policy Page.
Privacy Notice and Cookies 2019