Assessing and optimizing research and innovation impact: developing and implementing a holistic systems approach

Abstract

Increasing demand to demonstrate the societal impact of research and innovation investments comes not only from government and funders for accountability purposes, but also private industry and others, who want to use impact evaluation to inform ways of increasing benefits, and improve their return on investment. Citizens also wish to see an increase in benefits to their region. Impact assessments should also further the understanding of “what works or does not work, under what conditions for whom”. In Alberta, Canada, we developed a “fit for purpose” performance and impact management system that aims to assess impact and optimize returns for decision makers while engaging users. This paper describes the Alberta team’s work that evolved from applying an evidence-based practice framework for evaluating research impact. Drawing on various theories, methodological developments resulted from testing models and approaches across sectors for generating practice-based evidence. The paper outlines the evolution across six phases that contribute to the international literature, and highlights the development of applications, resulting in new learnings and reflections with each iteration. Key learnings are that designing for impact upfront, and continuously monitoring and evaluating impact are effective tools to inform adjustments along implementation pathways to enhance impact. Transitioning to a holistic systems approach that values quantitative and qualitative methods, and combines evidence from impact management systems is necessary for assessing challenge type initiatives. A promising practice is conducting complementary deep-dive realist impact assessment case studies to help understand “how” and “why” the impact occurred and who benefited.