Grant:
Institute of Education Services
Precision Education:
The Virtual Learning Lab, $8.9M
Grant Objective:
This 5-year project delves into precision education, researching how to capitalize on the large data sets accumulated by virtual learning systems to create truly personalized and highly efficient learning environments, ultimately improving student learning outcomes.
Project Overview:
Data from prior users of Algebra Nation, an online tutoring program developed by the University of Florida and Study Edge, is used to generate instructional sequences to optimize individual students’ learning outcomes. The five-year research plan includes two measurement studies, a pilot study with an experimental design in the third year, and an efficacy study in the fourth year. In addition to the focused program of research, the Virtual Learning Lab serves as a “hub” where researchers, practitioners, and policymakers collaborate on key issues around virtual learning.
Year 5: Summer 2020 – Spring 2021
June 2021
Research Showcase at the 2021 ED Games Expo
2021 Ed Games Expo
The ED Games Expo is an annual showcase of game-changing innovations in education technology that were developed through more than 30 programs at the US Department of Education and across the federal government. The full agenda of the Ed Games Expo is available here. Follow the ED Games Expo on twitter @USEDgov @SBIRgov and #EDGamesExpoAbout the ED Games Expo
The ED Games Expo is an annual showcase of game-changing innovations in education technology that were developed through more than 30 programs at the US Department of Education and across the federal government. Follow the ED Games Expo on twitter @USEDgov @SBIRgov and #EDGamesExpo
Check out the recording our presentation here.
April 2021
Event at the 2021 Meeting of the American Educational Research Association (AERA)
Year 4: Summer 2019 – Spring 2020
Spring 2020
Experimental Study of the Revised Video Recommendation Algorithm – February 3rd to May 31st, 2020
We recruited 74 teachers from Broward, Palm Beach and Duval school districts to study the improved video recommendation system for the Algebra Nation platform. We randomly assigned the ~14,000 students of the participating teachers to treatment or control conditions. The treatment group received video recommendations using the revised algorithm, while the control group was recommended to watch the next video in the Algebra Nation sequence. Due t COVID-19, the study had two periods: School-based instruction from February 3rd to March 17th; Online-based instruction from March 18th to May 31st. Preliminary results show that students who received the treatment condition during the first period of the study performed better in mathematics tests in Algebra Nation than students in the control group.
April 2020
Designing Online Platforms to Enable Research Event
Summer 2019
Supplemental Study
Through a partnership with Dr. Danielle McNamara at Arizona State University, 100 students in Broward County will be provided high quality, 1:1 tutoring in Algebra 1. Transcripts and EOC grades will be studied to understand what qualities create effective tutoring experiences.
Year 3: Summer 2018 – Spring 2019
Spring 2019
Virtual Learning Lab Panel at the Annual Meeting of the American Educational Research Association
Spring 2019
Personalization Algorithm Experimental Study 1
This semester, students taking Algebra 1 in Broward County Public Schools will experience the new and improved Video Recommendation Systems. Students will receive suggestions for the next video they should view, similar to the user experience in YouTube. One group will receive recommendations in the standard order, and one will receive recommendations based on their past performance. We’ll examine their EOC scores to see which system is more effective.
Spring 2019
Personalization Algorithm Experimental Study 1
This semester, students taking Algebra 1 in Broward County Public Schools will experience the new and improved Video Recommendation Systems. Students will receive suggestions for the next video they should view, similar to the user experience in YouTube. One group will receive recommendations in the standard order, and one will receive recommendations based on their past performance. We’ll examine their EOC scores to see which system is more effective.
November 2018
September 2018
New PI
Dr. Walter Leite assumes the role of PI for the rest of the project.
April 2018
Panel
A panel was held in conjunction with the annual American Education Research Association exploring opportunities and challenges of personalized learning in education settings.
April 2018
AERA Conference Re-taker Study poster presentation
The analysis examines the impact of Algebra Nation on the performance of students who failed the end-of-course (EOC) exam the previous year. Results indicate that higher use of Algebra Nation is associated with greater EOC gains, particularly for students most at risk for repeated failure.
Year 2: Summer 2017 – Spring 2018
January 2018
Initial Feasibility Study Launched
The recommendation version of Algebra Nation was pilot-tested in Spring 2018 in Alachua County Schools located in Gainesville FL. Students (N = 2,171) used the platform once per week during the spring semester. Teachers completed weekly implementation logs that were compared to usage reports and completed a post-study survey.
January 2018
IES Principle Investigator Meeting
VLL team hosts a panel on personalized learning, discussing positive and negative components. Topics included the impact on teachers, the costs of providing expanding content, etc.
Fall / Winter 2017
Measurement Study 2 Progress
Large-scale affect surveys were collected from Algebra 1 students from September-December 2017 (N= 25,629). Each survey prompt included a random question on one of 18 affective states in a non-obtrusive, light-touch approach. The data were used to train neural networks to predict students’ reports of their affective states from their activity with Algebra Nation in the previous 5 minutes. Performance was significantly above random baselines for 10 affective states: confusion, contentment, disappointment, engagement, frustration, happiness, hopefulness, pleasantness, pride and relief.
Fall 2017
Completion of Measurement Study 1
This analysis focused on the impact of Algebra Nation use on student’s scores on the end of course Algebra 1 exam (EOC). For the 2015-2016 school year, data from more than 80,000 students indicated that Algebra Nation users had better EOC performance than non-users.
July 2017
First Graduate Student Training Workshop
Students spent four full days on campus working with project members on topics such as a comparison of urban, suburban and rural schools, and an investigation of video tutors’ popularity as a function of gender, age and ethnicity.
Year 1: Summer 2016 – Spring 2017
Winter 2016
Measurement Study 2 Initiated
The team at CU leads research to develop sensor-free engagement estimates that can be used to assess the impact of personalized instruction. In addition, student reports of cognitive and emotional states are aligned with log files of on-task and off-task behaviors, with the goal of identifying “trigger events” that could be leveraged to promote engagement in an adaptive version of the system.
Fall 2016
Measurement Study 1 Initiated
The first study was initiated to identify a) usage patterns at the student level that are linked to EoC outcomes that can be used to drive a recommendation engine and b) usage patterns at the classroom level that are linked to EoC outcomes that can be used to design improved professional development.
Summer / Fall 2016
Build a Data Archive
The first step in the focused program of research was to build an archive of data from prior users to identify patterns of use that are linked to successful outcomes, using Algebra 1 End of Course exam scores, provided by the Florida Department of Education, as well as user data from the Algebra Nation platform.
July 2016
University of Florida
Dr. Carole Beal (UF), supported by Dr. George Michailidis (UF), Dr. Donald Pemberton (UF), Dr. Sidney D’Mello (CU), and Walter Leite (UF), secured $8.9M from the Institute of Education Sciences to support their research titled “Precision Education: The Virtual Learning Lab.”