Data Sources and Collection Methods - Canadian Association of Research Libraries

Introduction

“Management consultants frequently point out that ‘what gets measured is what gets done.’ Critics fear that this means organizations will be misled by metrics. To avoid this outcome, leaders are faced with the obligation of determining at the highest level the key questions their organizations must confront, rather than just react to those items easiest to measure and track” (Marcum & Schonfeld, 2014, p. 3).

Key Books and Reports

Connaway, L. S., Harvey, W., Kitzie, V., & Mikitish, S. (2017). Academic library impact: Improving impact & essential areas to research. Chicago: ACRL.

Summarizes the incredible strides made and best practices developed by the profession in capturing and emphasizing academic libraries’ contributions to student learning, success, and experience.

Farney, T. (2018). Using digital analytics for smart assessment. Chicago, ALA Editions.

Your library collects massive amounts of data related to this journey—probably more than you realize, and almost certainly more than you analyze. Farney shows you how to maximize your efforts: you’ll learn how to improve your data collection, clean your data, and combine different data sources.

Franklin, T., Harrop, H., Kay., D., & van Harmelen, M. (2011). Exploiting activity data in the academic environment. Retrieved from the Activity Data website.

This web site synthesises the work of the JISC funded activity data programme in order to help you to understand how you might benefit from exploiting activity data. Includes a special section on library data sources.

Marcum, D., & Schonfeld, R. C. (2014). Driving with data: A Roadmap for evidence-based decision making in academic libraries. New York, NY: Ithaka S+R.

COUNTER-compliant usage statistics, service assessments, peer benchmarking—librarians have been gathering different types of data for some time, using data to measure the usage of their resources, the quality of their services, and how they stack up against similar institutions. But could library leaders collect data differently? In this Issue Brief Deanna Marcum and Roger Schonfeld suggest an approach where library leaders start not with the data that are easy to gather, but with the problems they are trying to solve. What does it take to create an evidence-based decision-making environment within the academic library?

Orcutt, D. (2009). Library data: Empowering practice and persuasion. Santa Barbara, CA: Libraries Unlimited.

Numerical evidence is everywhere and how best to handle and leverage it is a growing concern in the academic world in general and the academic library world in particular. Libraries are not only storehouses and key contacts for library patrons in accessing numbers, but are also collectors and users of their own data, which is integral to the functioning of the library itself.

Key Points - Data Collection

What key performance indicators (KPIs) have been identified as important? How do they relate to our academic institution’s key strategic priorities?
Do we have the organizational structure, technical skills, and system in place to assemble, manage, and analyze the data?
Do we have the data we need? Do our systems and processes collect it? How much time and expense will be required to collect evidence?
If we don’t have the data, how do we start collecting it? Can we get it from somewhere else in the academic institution? What are the risks and what are the benefits of gathering evidence from external stakeholders?
Is there enough of it to make any sense? How long have we been collecting data and how much data is collected per year? What are the risks and benefits of simply making the decision oneself or internally?
Should we combine a number of data sources to paint a fuller picture? If so, are there reliable data elements held in common across the data sources to make combination possible (e.g. Student ID number)? If not, can we begin adding this data element to a data set to make linkage possible?
Will the analysis serve the analytical purpose (i.e. provide KPI) we have in mind? Or could it trigger new analyses?
Remember that data, like statistics, can lend itself to different interpretations. What does “high website traffic really mean”?
- the site is valuable and used?
- the same few people make high use of it but most people don’t use it?
- people are trying different things because they can’t find what they are looking for?
Data can be quantitative or qualitative. Data can be pre-existing or can be gathered to meet a specific need. Data can be at the local level, whether branch, library system, or institution, or can be at a regional, national, or association level.
There are many different methods of collecting data, from manual collection of tick marks on a sheet of paper to automated downloads from an online source, from formal research to published reports.
Automated data can be acquired by manual data entry, by capturing data during an activity (i.e. circulating a book captures data on the book, the user, the time period), or by data logs generated as traffic moves though a system (i.e web page views).
There are many different programs that can be used to compile and manipulate your data, from simple calculations on a sheet of paper, to excel spreadsheets or more data intensive statistical programs like SPSS or SAS, from analytic modules within your library services software to purchased data services.

Benchmarking

“Many academic institutions use benchmarks to identify their strengths and weaknesses in comparison to similar institutions. For example, benchmarking can be used to demonstrate whether an institution or its library is funded or staffed at levels comparable to similar institutions in a geographic area, with a similar enrollment, or with other related characteristics. An institution or library can use benchmarking to inform the strategies it develops to enhance its institutional quality and effectiveness” (ACRL, 2018, p. 21).

Basic Examples

Institutional expenditure on library relative to peers
Number of students and faculty relative to peers
Library expenditures per student and faculty relative to peers
Collection budgets (physical and electronic) relative to peers
Online resources (journals, books, and databases) relative to peers
Number of library staff relative to peers

More Complex Example

Lewin, H. S., & Passonneau, S. M. (2012). An analysis of academic research libraries assessment data: A look at professional models and benchmarking data. The Journal of Academic Librarianship, Volume 38(2), 85-93.

Benchmarking Data Sources

Key Report

ACRL. (2018). Standards for Libraries in Higher Education. Chicago, IL: ACRL.

The Standards for Libraries in Higher Education are designed to guide academic libraries in advancing and sustaining their role as partners in educating students, achieving their institutions’ missions, and positioning libraries as leaders in assessment and continuous improvement on their campuses. Libraries must demonstrate their value and document their contributions to overall institutional effectiveness and be prepared to address changes in higher education, including accreditation and other accountability measures.

Collections & Resources

Basic Examples

Number of titles (physical + electronic)
Total library materials expenditures percentages. E.g.: Monograph expenditures as percentage of total library materials expenditures
Materials expenditures to total library expenditures percentages. E.g.: Total library materials expenditures as percentage of total library expenditures
Total library materials expenditures per student. E.g.: Per full-time undergraduate student. Per full-time graduate student.
Total library materials expenditures per faculty. E.g.: Per full-time faculty. Per part-time faculty

More Complex Example

Correlate growth in electronic library material expenditures to growth in research productivity.
See for example: Rawls, M. M. (2015). Looking for links: How faculty research productivity correlates with library investment and why electronic library materials matter most. Evidence Based Library & Information Practice, 10(2), 34-44.

Key Report

Harker, K. R., & Klein, J. (2016, September). Collection assessment. SPEC Kit 352. Washington, DC: Association of Research Libraries.

This study gathered information on which library staff collect and analyze data, for what purposes the results are used, with whom data is shared, how well assessment questions are answered. The survey also investigated whether the available methods, data, and tools are aligned with the purposes for assessing collections.

Discovery & Delivery

“Libraries enable users to discover information in all formats through effective use of technology and organization of knowledge” (ACRL, 2018, p. 9).

Basic Examples

Number of users accessing library’s web site
Number of users accessing licensed electronic resources: COUNTER provides the Code of Practice that enables publishers and vendors to report usage of their electronic resources in a consistent way. This enables libraries to compare data received from different publishers and vendors.
Number of items downloaded
Number of book loans

More Complex Example

Correlate usage of library resources and academic peformance. See for example: Janetti, M., & Cox, B. (2013). Measuring the value of library resources and student academic performance through relational datasets. Evidence Based Library and Information Practice, 8(10), 163-171.

Key Report

MINES for Libraries: As libraries implement access to electronic resources through portals, collaborations, and consortial arrangements, the Measuring the Impact of Networked Electronic Services (MINES) for Libraries protocol offers a convenient way to collect information from users in an environment where they no longer need to physically enter the library in order to access resources.

Education & Instruction

“Libraries partner in the educational mission of the institution to develop and support information-literate learners who can discover, access, and use information effectively for academic success, research, and lifelong learning” (ACRL, 2018, p. 9).

Basic Examples

Number of group training sessions (physical and virtual) offered
Number of attendees (physical and virtual) at group training sessions
Number of courses in which librarians are embedded
Number of students reached through embedded information literacy initiatives

More Complex Example

Correlate the number of participants in library instruction classes to student retention. See for example: Eng, S. (2015). Linking library to student retention: A statistical analysis. Evidence Based Library & Information Practice, 10(3), 50-63.

Key Report

Oakleaf, M. (2010). Value of academic libraries: A comprehensive research review and report. Chicago, ACRL.

The ACRL publication Value of Academic Libraries: A Comprehensive Research Review and Report is a review of the quantitative and qualitative literature, methodologies and best practices currently in place for demonstrating the value of academic libraries, developed for ACRL by Megan Oakleaf of the iSchool at Syracuse University.

Spaces & Places

“Libraries are the intellectual commons where users interact with ideas in both physical and virtual environments to expand learning and facilitate the creation of new knowledge” (ACRL, 2018, p. 9).

Basic Examples

Ratio of library seats to FTE student population
Type of learning spaces and accompanying technology available to user community
Number of hours open each week during academic sessions
Number of days open each fiscal year
Gate counts. E.g.: Per FTE student. During extended hours of fall and spring semesters.

More Complex Example

Correlate library usage to GPA – the library’s influence appears through two factors that highlight the library as a place including providing a place to study alone and as a place that has specialized equipment available to students. See for example: Stemmer, J. K., & Mahan, D. M. (2015). “Assessing the library’s influence on freshman and senior level outcomes with user surveys”. Evidence Based Library & Information Practice, 10(2), 8-20.

Key Report

Lippincott, J. K., & Duckett, K. (2013). Library space assessment: Focus on learning. Research Libraries Issues: A Report from ARL, CNI, and SPARC, (284), 21-21.

The “Library Space Assessment: Bringing the Focus to Teaching and Learning” 3 workshop, held at the Library Assessment Conference in November 2012, was designed to help participants think more deeply about connecting completed space renovation assessment to student learning.