Predictive dashboard elements
- 1. Adding predictive elements to student and instructor dashboards Robert Bodily Brigham Young University
- 2. Context • Class • First year chemistry course • Blended – class 3x per week, • Resources • 150 videos (avg. 2 min long, supplemental resources) • 15 weekly quizzes (unlimited question attempts) • Participants • 200 students (online interactions) • 96 students took the self-report resource use survey
- 3. Nature of the data • Quiz • Confidence in answer (just a guess, pretty sure, very sure) • Time spent on quiz • Correct/incorrect • Number of attempts per question • Leave tab (still open, but inactive), come back to tab (active again) • Video • Play, pause, skip forward/backward, change play rate, change volume, • Dashboard • Number of times students follow recommendations given in dashboard • Number of clicks within the dashboard
- 6. Goals Our dashboards are completely descriptive, so I want to add predictive elements to both the student and instructor dashboards 1. Understand what course elements are predictive of student achievement (grade on final exam) 2. Develop an early course prediction of student success 3. Determine what student profiles exist based on online behavior 4. Develop a model to classify future students into groups
- 7. What course elements are predictive of student success? Variable Beta P-value Online homework score 0.366 0.000 In-class IClicker scores 0.154 0.024 # of attempts/question -0.411 0.000 Amount of question navigation -0.206 0.040 # of online activity sessions -0.195 0.020 Variable Beta P-value Read the textbook 2.443 0.059 Ask professor questions in class 7.363 0.000 Watch Khan Academy -2.738 0.051 Use the internet -3.199 0.010 Skip recitation -4.820 0.041 Model 1 – regressing online interaction data on final exam score. Model 2 – regressing self-report resource use on final exam score.
- 8. Develop an early course prediction of student achievement Online student interaction data Online student interaction data AND exam scores There is significant improvement in both models until week 3 or 4, so that seems to be a good time to make predictions for instructors and students.
- 9. Clustering to find student profiles Cluster 1: Higher prior knowledge, good study skills, uses email, does not use office hours Cluster 2: Efficient, game the system, ok losing some points, office hour students, does not use email Cluster 3: Work hard but inefficiently, use tutor/friends, low self-regulation, bad study habits 3
- 10. Develop a classification model to classify future students into groups Classes AIC BIC SSA BIC Log Likelihood 2 8299 8460 8305 -4101 3 7870 8086 7877 -3869 4 7502 7774 7511 -3668 5 7139 7467 7150 -3470 6 6813 7196 6826 -3290 7 6612 7051 6626 -3172 8 6523 7017 6539 -3110 Group counts 3. 2 1. 20 2. 119 5. 24 6. 8 4. 22 Group 3: efficient smart learners Group 1: low online activity, efficiency driven, just getting by Group 2: average students, lower effort, low online activity Group 5: average students, high effort, high online activity Group 6: low learning skills, low knowledge awareness, high effort Group 4: low learning skills, low knowledge awareness, but put forth less effort than group 6
- 11. Student dashboard suggestions • Provide students with recommendations on the things that good students do to succeed in the course, as well as the potential effect of these things. • Give students feedback on how they will do on the final based on historical students similar to them. Give them something to click to act on this information (e.g. meet with TA, meet with instructor, etc.). • Show students some examples of their online behaviors to make them more aware of their online activity. Provide recommendations to help them improve.
- 12. Instructor dashboard suggestions • Provide a list of things successful students do along with their effect on student final exam grade so the instructor can encourage students to do them to improve in the course • Provide a predicted pass/fail score for each student at week 3 or 4 in the course so the instructor or teaching assistants can intervene with potentially struggling students • Provide a student profile for each student so the instructor can better personalize feedback to students