Grant:

Institute of Education Services

Precision Education:

The Virtual Learning Lab, $8.9M 

Grant Objective:

This 5-year project delves into precision education, researching how to capitalize on the large data sets accumulated by virtual learning systems to create truly personalized and highly efficient learning environments, ultimately improving student learning outcomes.

Project Overview:

Data from prior users of Algebra Nation, an online tutoring program developed by the University of Florida and Study Edge, is used to generate instructional sequences to optimize individual students’ learning outcomes. The five-year research plan includes two measurement studies, a pilot study with an experimental design in the third year, and an efficacy study in the fourth year. In addition to the focused program of research, the Virtual Learning Lab serves as a “hub” where researchers, practitioners, and policymakers collaborate on key issues around virtual learning.

Year 5: Summer 2020 – Present

June 2021

ED21 Games Expo Research Showcase flyer

Research Showcase at the 2021 ED Games Expo

April 2021

American Educational Research Association (AERA) VLL flyer

Event at the 2021 Meeting of the American Educational Research Association (AERA)

Summary Poster of the Event

Year 4: Summer 2019 – Spring 2020

Spring 2020

Experimental Study of the Revised Video Recommendation Algorithm – February 3rd to May 31st, 2020

We recruited 74 teachers from Broward, Palm Beach and Duval school districts to study the improved video recommendation system for the Algebra Nation platform. We randomly assigned the ~14,000 students of the participating teachers to treatment or control conditions. The treatment group received video recommendations using the revised algorithm, while the control group was recommended to watch the next video in the Algebra Nation sequence. Due t COVID-19, the study had two periods: School-based instruction from February 3rd to March 17th; Online-based instruction from March 18th to May 31st. Preliminary results show that students who received the treatment condition during the first period of the study performed better in mathematics tests in Algebra Nation than students in the control group.

April 2020

Algebra Nation Logo

Designing Online Platforms to Enable Research Event

Summer 2019

Arizona State University

Supplemental Study

Through a partnership with Dr. Danielle McNamara at Arizona State University, 100 students in Broward County will be provided high quality, 1:1 tutoring in Algebra 1. Transcripts and EOC grades will be studied to understand what qualities create effective tutoring experiences.

Year 3: Summer 2018 – Spring 2019

Spring 2019

American Educational Research Association Meeting Flyer

Virtual Learning Lab Panel at the Annual Meeting of the American Educational Research Association

Spring 2019

Personalization Algorithm Experimental Study 1

This semester, students taking Algebra 1 in Broward County Public Schools will experience the new and improved Video Recommendation Systems. Students will receive suggestions for the next video they should view, similar to the user experience in YouTube. One group will receive recommendations in the standard order, and one will receive recommendations based on their past performance. We’ll examine their EOC scores to see which system is more effective.

Spring 2019

Personalization Algorithm Experimental Study 1

This semester, students taking Algebra 1 in Broward County Public Schools will experience the new and improved Video Recommendation Systems. Students will receive suggestions for the next video they should view, similar to the user experience in YouTube. One group will receive recommendations in the standard order, and one will receive recommendations based on their past performance. We’ll examine their EOC scores to see which system is more effective.

September 2018

Dr. Walter Leite

New PI

Dr. Walter Leite assumes the role of PI for the rest of the project.

April 2018

Algebra Nation Logo

Panel

A panel was held in conjunction with the annual American Education Research Association exploring opportunities and challenges of personalized learning in education settings.

April 2018

The Dreams, Possibilities, and Necessity of Public Education: 2018 Annual Meeting

AERA Conference Re-taker Study poster presentation

The analysis examines the impact of Algebra Nation on the performance of students who failed the end-of-course (EOC) exam the previous year. Results indicate that higher use of Algebra Nation is associated with greater EOC gains, particularly for students most at risk for repeated failure.

Year 2: Summer 2017 – Spring 2018

January 2018

Alachua County Public Schools

Initial Feasibility Study Launched

The recommendation version of Algebra Nation was pilot-tested in Spring 2018 in Alachua County Schools located in Gainesville FL. Students (N = 2,171) used the platform once per week during the spring semester. Teachers completed weekly implementation logs that were compared to usage reports and completed a post-study survey.

January 2018

Annual Principal Investigators Meeting graphic

IES Principle Investigator Meeting

VLL team hosts a panel on personalized learning, discussing positive and negative components. Topics included the impact on teachers, the costs of providing expanding content, etc.

Fall / Winter 2017

Measurement Study 2 Progress

Large-scale affect surveys were collected from Algebra 1 students from September-December 2017 (N= 25,629). Each survey prompt included a random question on one of 18 affective states in a non-obtrusive, light-touch approach. The data were used to train neural networks to predict students’ reports of their affective states from their activity with Algebra Nation in the previous 5 minutes. Performance was significantly above random baselines for 10 affective states: confusion, contentment, disappointment, engagement, frustration, happiness, hopefulness, pleasantness, pride and relief.

Fall 2017

Algebra Nation Logo

Completion of Measurement Study 1

This analysis focused on the impact of Algebra Nation use on student’s scores on the end of course Algebra 1 exam (EOC). For the 2015-2016 school year, data from more than 80,000 students indicated that Algebra Nation users had better EOC performance than non-users.

July 2017

Institute of Education Sciences Logo

First Graduate Student Training Workshop

Students spent four full days on campus working with project members on topics such as a comparison of urban, suburban and rural schools, and an investigation of video tutors’ popularity as a function of gender, age and ethnicity.

Year 1: Summer 2016 – Spring 2017

Winter 2016

Student with headphones

Measurement Study 2 Initiated

The team at CU leads research to develop sensor-free engagement estimates that can be used to assess the impact of personalized instruction. In addition, student reports of cognitive and emotional states are aligned with log files of on-task and off-task behaviors, with the goal of identifying “trigger events” that could be leveraged to promote engagement in an adaptive version of the system.

Fall 2016

Algebra Nation Logo

Measurement Study 1 Initiated

The first study was initiated to identify a) usage patterns at the student level that are linked to EoC outcomes that can be used to drive a recommendation engine and b) usage patterns at the classroom level that are linked to EoC outcomes that can be used to design improved professional development.

Summer / Fall 2016

Florida Department of Education Logo

Build a Data Archive

The first step in the focused program of research was to build an archive of data from prior users to identify patterns of use that are linked to successful outcomes, using Algebra 1 End of Course exam scores, provided by the Florida Department of Education, as well as user data from the Algebra Nation platform.

July 2016

Institute of Education Sciences Logo

University of Florida

Dr. Carole Beal (UF), supported by Dr. George Michailidis (UF), Dr. Donald Pemberton (UF), Dr. Sidney D’Mello (CU), and Walter Leite (UF), secured $8.9M from the Institute of Education Sciences to support their research titled “Precision Education: The Virtual Learning Lab.”