RESEARCH AND DATA POLICIES AND PRACTICES

Improving Learning Outcomes

The Open Learning Initiative (OLI) at Carnegie Mellon University strives to transform teaching and learning by leveraging learning science and emerging technologies to increase student success, improve learning productivity and lower barriers to access. By rigorously capturing and evaluating learner data, OLI is able to drive powerful feedback loops that assist learners and educators, improve courses and contribute our larger understanding of how humans learn.

Commitment to Research

OLI’s work leverages research in cognitive and learning science while enacting research to discover new insights into how humans learn. This combination of research and practice is an essential part of the organization and reflects CMU’s pioneering learning engineering approach.

OLI’s research and data practices are reflected in the Asilomar Convention for Learning Research in Higher Education, which asserts the following tenets for learning research:

1. We are committed to advancing the science of learning for the improvement of higher education.

“The science of learning can improve higher education and should proceed through open, participatory, and transparent processes of data collection and analysis that provide empirical evidence for knowledge claims.” (Asilomar Convention)

2. We are committed to sharing data, findings and technologies within the learning research community in order to extend the project’s contributions to learning science.

“Maximizing the benefits of learning research requires the sharing of data, discovery, and technology among a community of researchers and educational organizations committed, and accountable to, principles of ethical inquiry held in common.” (Asilomar Convention)

OLI seeks to advance learning science by discovering and sharing insights about learning and how to improve learning using data collected through courseware as well as related learner data from institutions. We will adhere to transparent, responsible and ethical practices around data ownership, sharing and use. OLI is also committed to compliance with institutional, state and federal policies regarding appropriate handling and use of learner data.

Consent for Use of Data

In order to function fully and effectively, OLI’s courseware captures and uses a variety of instructor and learner data within the system for which no explicit permission is required beyond using the courseware. The courseware uses learners’ own data to help them understand their learning process and progress. The courseware shares learner data with instructors to provide visibility into student behaviors and learning so that instructors can facilitate more effective learning. OLI uses these data to evaluate and improve the effectiveness of its courseware in supporting student learning.

We require consent from students and instructors to use their learning data for research purposes. Implemented with review, approval process oversight from CMU’s Institutional Review Board (IRB), this approach uses an opt-in/opt-out form to confirm user consent for authorized researchers and research communities to use their de-identified data in research studies. Students may opt in or opt out repeatedly, allowing them to change their minds about participation at any point.

Data Collection

OLI’s courseware captures comprehensive data that is meaningfully contextualized with semantic markup. These measures strengthen data quality, analytical capability, and searchability. Contextualized data allows us to conduct a full spectrum of analyses and discovery in support of student learning. It future-proofs our work by providing the means to explore new questions or develop new data models as our understanding of the courseware and learning science evolves. Our courseware captures and uses the minimal personally identifiable information (PII) elements of name and email address in order to facilitate a variety of functions within the courseware; this PII is not available for research under our ongoing IRB protocol.

Uses of Data, Data Models, and Analysis

OLI seeks to identify the most important feedback loops around improving student learning, including the use of data to encourage metacognition and to expose and reinforce study behaviors that lead to increased learning. Data capture, research and analysis focus on:

• Feedback loops for students
• Feedback loops for faculty
• Feedback loops for course design

Current learning science informs our initial hypotheses about these feedback loops. Over time our hypotheses and research strategies evolve along with the courseware and our understanding of its impact on learning. Data analyses drive continuous iterative improvements, accompanied by success measurements to gauge efficacy.

OLI researchers compile research questions to drive courseware instrumentation for data collection, research methodologies, and data model development. To address these questions, we apply a variety of analytical techniques to better understand and improve student learning including techniques such as:

• Learning analytics: Analytics that validate learning has taken place
• Engagement analytics: Analytics documenting measurable activities such as levels of interaction with content, what happens in the classroom, personal interaction, etc.
• Progression analytics: Analytics that gauge movement through a course and/or an education program over time
• Courseware analytics: Analytics that establish how well courseware is supporting student learning

Wherever possible and appropriate, courseware design and research data models support variability and divergent pathways for students to achieve success, rather than “one size fits all.” We employ multiple data models such as:

• Cognitive models: How are students learning effectively?
• Adaptation models: What approaches and practices will better support the learner? The instructor? Courseware efficacy?
• Assessment models: What types of assessments are most effective at demonstrating mastery of learning objectives?
• Iterative improvement models: What is most effective in facilitating continuous improvement in the courseware?

OLI makes its research findings publicly available in order to broaden the impact of our work on education and learning science, while maintaining privacy protections for learners.

Security, Risk and Liability

OLI and its partners employ best practices around information security to ensure minimal collection of personally identifiable data and ensure what PII is collected remains secure. These practices impact courseware architecture and functionality as well as the behaviors of learners, instructors, institutional staff, and the courseware provider.

Data Architecture, Systems and Technologies

OLI leverages the capabilities of LearnLab’s DataShop and Learnsphere tools to de-identify data and make that de-identified data available to authorized learning research communities in accordance with IRB requirements.

Continuous Consideration of Research and Data Practices

We review our strategies, policies, and practices around data and research at least annually to assure that they align with recommended standards among researchers, learners and educational institutions.

Collaborative with Partners

OLI collaborates extensively on research with partner institutions. As part of these collaborations we may collect and analyze additional data in partnership with the institution(s). These data are handled according to the principles and processes described above. Any research outside these practices are done only with oversight and review by the CMU and/or partner IRB.

Additional Questions?

Additional questions about OLI’s research and data practices can be directed to OLI Help or to OLI’s Principal Investigator and Director Norman Bier, at nbier@cmu.edu.

Last updated 03/27/2020