CARL’s Library Impact Framework Project was initiated in 2019 as a project of CARL’s Assessment Committee. While there are numerous projects and initiatives on the topic of library impact (ARL, ACRL, etc.), the CARL Assessment Committee recognized the question of how we frame library impact as a ‘wicked problem’ in need of further reflection, study, and examination. That is to say, framing library impact is complex, elusive, and methodologically fraught, but also meaningful, valuable, and insightful. Our aim was to bring a fresh perspective to the problem by taking a holistic approach that would be inclusive of diverse quantitative and qualitative approaches.
In this work, the CARL Assessment Committee was inspired by a recent Federation for the Humanities and Social Sciences (FHSS) report on the impact of research in the humanities and social sciences. The FHSS report provides a framework for thinking about impact in research that acknowledges diverse outcomes that defy easy measurement and argues that impact may be attributed to the collective work of scholarship playing out over long timeframes. The FHSS report employs the concept of “impact pathways” – complex relationships between inputs, outputs and outcomes that may be thought of as arcs of influence that cannot be meaningfully rendered using bibliometrics or causal indicators.
The CARL Library Impact Framework Working Group saw in this report a useful parallel for our attempts to describe and demonstrate impact within our libraries. Purely quantitative measures and studies of correlation (often framed as causal) do not do justice to the impacts of our libraries on teaching, learning and research within our institutions and beyond. Conversely, purely qualitative measures and studies are often seen as anecdotal or ungeneralizable.
Logic Models
This CARL Library Impact Framework has taken the idea of impact pathways and used logic models as a way to visualize the arc of influence of our libraries’ programs, resources, and services. Logic models are predominantly used for the purposes of program planning and evaluation in the social services, government, and not-for-profit sectors. They have been used less frequently in the academic library context.
The logic model framework has the benefit of:
- Providing a structured approach
- Delineating input from output from impact
- Aiding us in describing the different forms impact can take
- Articulating where we believe we have impact
- Assisting us in uncovering the assumptions we have about our impact
We have chosen to adopt a fairly standard logic model template with one important modification. The bottom half of a standard logic model template is intended for assessment strategies that align with each column (input, output, outcomes). In this section we are overlaying a gradient that runs from evidence to insights as one moves from inputs on the left towards outcomes and impacts on the right. This is intended to acknowledge the legitimate role of qualitative or interpretative methodologies in library impact assessment, where so often the focus is on quantitative indicators and the ‘holy grail’ of causal evidence of impact. The gradient from evidence to insight is not intended to apply in every instance, but to give proper recognition of interpretation in impact discourse which tends to be more applicable when describing outcomes.
Logic models are, however, not without their limitations. One limitation is that it appears as a linear model which may visually suggest that planning processes begin with input and proceed toward output and outcome – whereas it may be more appropriate to depict planning processes as proceeding in the reverse order from mission to strategy to program to input. A more accurate rendering might be circular or at least bi-directional. Despite these reservations, we chose to use the standard logic model for practical reasons since a more fluid or circular model would be challenging to use as a template.
For more on logic models, see Enhancing Program Performance with Logic Models, University of Wisconsin-Extension, February 2003.
How to Use the Framework
The CARL Library Impact Framework is offered as a set of exemplars intended to demonstrate how logic models may be applied to library programs, services, and resources. The logic models in the Framework are not intended to be exhaustive or complete in themselves. While the logic models presented here may be useful as stand-alones, the value is primarily in the application of the model, and in the process of thinking through logic models for local needs. For instance, a logic model created by an institution on an institutional repository may look different from the one contained within the framework due to local circumstances and context. Additionally, a strength of the logic model is that it can be used for the assessment of micro-level activities (e.g., an individual librarian’s instructional program), as well as, for macro-level initiatives (e.g., structured integration of library instruction in first year courses).
Uses of the logic model framework include:
- The logic model approach may be used in mapping out assessment strategies when developing or evaluating programs. The rigour of the model can help bring clarity to assessment strategies.
- The structure and rigour of the logic model will also be useful in framing research questions related to impact. For instance, logic models may inform a research agenda by helping identify where impact has been documented and where it has not, or what assumptions are made about impact in a particular area.
- The framework may also be used in further professional development in assessment methods (e.g., in CARL’s Canadian Library Assessment Workshop).
- As a framework to collate, collect and share CARL impact projects/stories.
The hope is that this project will not be static but will be a living framework that can grow and expand as this important work is taken up by members of CARL institutions and the Canadian library assessment community.
Notes on the Logic Model Template
Institutional Priority & Library Mandate
Each of the logic model exemplars in the Framework has a header that situates each program under the headings of Institutional Priority and Library Mandate. Institutional Priority is intended to suggest the broad area of institutional strategy associated with the program. These are common themes within most institutions’ strategic plans: Student Learning & Experience; Research, Scholarship & Creative Activity; and Community Engagement & Outreach. The Library Mandate is intended to suggest the general kind of library function. These categories commonly figure within library mission statements: Resources, Services, and Spaces. Taken together these can form a matrix which can be used to plot different kinds of library programs and their general area of institutional impact.
| Student Learning & Experience | Research, Scholarship & Creative Activity | Community Engagement & Outreach | |
| Resources | Open educational resources | Collections in support of research | |
| Services | Library instruction | Institutional repository | |
| Spaces | Library learning spaces |
Outcomes
The logic model template used in this Framework includes three columns under Outcomes & Impact labelled Short-Term Results, Intermediate Results, and Long-Term Results. While these labels suggest a distinction based on temporal proximity, we have chosen to overlay the categories of: Learning, Action, and Condition. Accordingly:
- Short-term outcomes are framed as relating to learning, namely effecting changes in awareness, knowledge, attitude, skills, opinions, motivations, or aspirations.
- Intermediate outcomes are framed as relating to actions, namely changes in behaviour, practice, decision-making, etc.
- Long-term outcomes are framed as relating to changes in conditions, such as social, economic, or developmental conditions. These impacts tend to be further out and admittedly more challenging to measure.
Recognition
Special thanks to the members of the Library Impact Framework Project Team: Justine Wheeler (Co-Chair, University of Calgary), Tania Gottshalk (Co-Chair, Thomson River University), Carisa Polischuk (University of Saskatchewan), Kristen Romme (Memorial University), Tristan Muller (Université du Québec à Montréal) and Denis Ouellette-Roussel (Université du Québec à Montréal); and to the Impact Framework Working Group members: Mark Robertson (Chair, Brock University), Mary-Jo Romaniuk (University of Calgary), Shailoo Bedi (University of Victoria), and Julie Morin (CARL). Thanks also to the CARL Assessment Committee for support of this project.
