Data Back: Formative Assessment Design & Analysis for EcoMOBILE
I've had the pleasure of working with some great people this semester in HGSE's T561 (Transforming Education through Emerging Technologies, taught by Chris Dede), and I'd like to share my independent research paper below:
Data Back: Formative Assessment Design & Analysis for EcoMOBILE
Jared B. Fries
May 4, 2015
Submitted for partial fulfillment of HGSE T561
An extension of the EcoMOBILE team assignment
Transforming Education through Emerging Technologies
Prof. Chris Dede
Abstract:
EcoMOBILE has the potential to give a richer learning experience through the use of augmented reality in an ecosystem fieldtrip, but its design could be improved in order to overcome limitations in teaching, learning, and research. Most notably, survey fatigue and delayed impact constrain the effectiveness of the EcoMOBILE intervention. As such, this study outlines an output design for a stealth formative assessment (“data back”) within the EcoMOBILE learning experience in order to centralize data collection and further benefit student achievement through personalized recommendations.
Background and Purpose:
Current EcoMOBILE research focuses on three treatments (unprompted, prompted, and augmented) to evaluate the degree of augmented reality materials during a middle school ecosystem field trip. Students are first given a pre-survey before experiencing virtual reality ecosystem simulations in EcoMUVE. Learners then take a mid-survey before embarking on an ecosystem fieldtrip to a local pond using an augmented reality interface with EcoMOBILE. Following the field trip, the teacher initiates a discussion and reflection before students end the experience with a post-survey.
While this study is vital in identifying proper technology interventions for student engagement, student achievement, and student exposure to STEM careers, current limitations in its design limit its impact. First, survey fatigue could affect both the quality of data for research and the holistic student experience. Three nearly identical surveys seek to measure the effect of each intervention, yet an embedded formative assessment measure within EcoMOBILE has the potential to serve as a stealth assessment for one of the surveys, thus reducing survey fatigue while saving student time. Secondly, the effects of the research currently benefit future iterations, delaying the impact on student learning until the following year. Empowering the teacher with data immediately after, or even during, the fieldtrip has the potential to improve both the affective experience and content acquisition through adaptive reflective discussions and personalized interventions.
As such, this study seeks to highlight the features and benefits of a data back design for EcoMOBILE. While there are technical limitations in implementing this design[1], this paper looks at the theoretical potential for this emerging technology to offer a more transformative learning experience. Much of the inspiration from this paper draws on the work of Val Shute[2], as well as the educational technology company Clever’s current initiatives. Thus, the research question this study will address is, “How can formative assessment measures in EcoMOBILE further assist teaching and learning?”
Methodology:
The author was tasked within the EcoMOBILE research team to contribute to the qualitative analysis through transcribing GoPro videos, coding the transcripts, and evaluating additional measures from the videos not captured by log files (i.e. uni- vs. multi-lateral decision making). While this process was time intensive, the results contributed to an AERA paper in April 2014. Please refer to the mid-semester report for progress details and suggestions for improving the research assignment; this paper, however, will focus on the second half of the semester.
Following the assigned qualitative analysis assignment and empowered with a holistic understanding of the research questions and the student experience, the author designed a four-step approach towards designing a stealth assessment in the remaining four weeks of the semester. The first step was to define learning outcomes, before identifying points in EcoMOBILE that aligned to the objectives. Next, it was important to design the features and benefits of an output that would assist teachers in evaluating student performance. Lastly, the author considered which of the features would be most feasible and useful to teachers.
Assessable Outcomes:
There are three main learning goals of the ecosystem unit: affective, content, and process. The table below summarizes an assessment of assessable outcomes:
Figure 1: Assessing Assessable Outcomes
Student Goals during EcoMOBILE
Feasibility of Evaluating in Formative Assessment
1. Confidence in Content [Affective]
Flag timestamps and end-point as crude measures
2. Experience being a scientist [Affective/Process]
N/A
3. Student Engagement [Affective]
Flag significant deviations from screen time for crude, immediate feedback; transcripts are richer.
4. Terminology (Define) [Content]: i.e. Dissolved Oxygen (D.O), Respiration, pH, nitrates, turbidity, chlorophyll, photosynthesis, oxygenation, reproduction, decomposition
Yes, but detracts from exploration, which could impact #’s 2-3 [research on engagement with more output]
5. Terminology (Apply) [Content]: D.O. (visual identification), D.O. (locate expected values)
Yes – EverNote; Log files (could insert interactive touch-element for questions)
6. Recall EcoMUVE to describe simulations of place and time [Process]
Yes, but detracts from exploration, which could impact #’s 1-2. Could be more effective as reflection post-field- trip
7. Data Analysis comparing D.O. in 2 ponds [Process]
See above. There’s so much going on during a field trip and not all students got to both ponds.
The table above highlights important factors in designing the data back system. Affective measures (#’s 1-3) are extremely difficult to measure in a stealth assessment and might be best kept as self-reported data. Time stamps could be used as crude measures of affective experience; for instance, if a pair of students spends a significantly variant amount of time on or away from a particular screen, it could indicate whether or not the student is enjoying the experience, or perhaps is off-task. Content acquisition (#’s 4-5) are the most feasible to measure within the current user interface, but asking students for more output could detract from more affective measures. This is a question the EcoMOBILE team is considering researching in future years. While the learning process (#’s 6-7) – including analyzing and interpreting data, making predictions, and making observations - is one of the most important goals of the ecosystem unit, it most notably detracts from the affective experience and could be more useful as a reflection.
Entry Points for Formative Assessment:
Based on the analysis that content acquisition is easiest to measure as a stealth assessment, the following list highlights some potential areas within the prompted (“Bluegills”) treatment where formative assessment is possible, and which output (log file or Evernote) could best measure student input.
Applying photosynthesis with visual evidence (Bluegill 10 - Evernote)
Applying D.O. with visual evidence of factors (Bluegill 14, 33 - Evernote)
Applying D.O. with expected values / comparing (Bluegill 35 – log file)
Terminology definition of D.O. (Bluegill 21 – Evernote)
Measure D.O. & temperature at Quinn Pond (Bluegill 26-7 log file) and Dolan Pond (39-40 log file)
Figure 2: Bluegill 35
As illustrated above in Figure 2: Bluegill 35, it is feasible in the current EcoMOBILE program to insert formative assessment measures. The log file, if the software is designed correctly, could then illustrate the students’ answers. It is therefore feasible to assess whether or not students acquired the content, understood the learning process, and could make a prediction. While it is easiest to assess in questions with a binary response to gauge whether the students are right or wrong, a member of the research team suggested that a histogram of word cloud responses could help visualize open-ended questions such as the one displayed in Figure 2. Alternatively, the questions could be graded individually. As illustrated in the following section, this data could then be aggregated or individualized for further learning interventions.
Design Analysis:
What items in an output would assist teachers in evaluating student performance during the fieldtrip? Since knowledge acquisition is highly decentralized – students are learning in independent pairs far away from the teacher – the main goal of the data back design is to centralize assessable outcomes and to display the data analytics as rich visuals to inform pedagogical decisions. The chart below summarizes five suggested features and their subsequent benefits to teaching and learning:
Figure 3: Data Back Design
Features
Benefits
1. Embedded formative assessment
Saves student time in survey response and decreases survey fatigue.
2. Automatic question scoring
Saves teacher time in grading.
3. Aggregated performance by question (slide 6) or sub-genre (slide 5)
Teacher can spend time discussing where student content is weakest, or can pair students in heterogeneous groups.
4, Recommendations for individualized interventions
Plans interventions for students with weakest content acquisition.
5. Adaptive content for personalized learning plans
Tailors homework assignments or review questions to student weaknesses to prepare for final survey / exam. Could also scale tests for personalized assessment.
The previous sections discussed how an embedded formative assessment (Feature #1) is feasible within certain parts of EcoMOBILE, but a program that automatically scored the responses (Feature #2) could assist in saving the teacher’s time from entering scores in manually. Entering student data, whether automatic or manual, is a prerequisite for subsequent features.
Feature #3 centralizes the data and displays visuals for aggregated results (i.e. percent of students who could apply dissolved oxygen knowledge based on responses to a certain question) or disaggregated performance (i.e. which student groups performed weakest in finding an example of photosynthesis in nature). Currently, the data from the log file or the mid-survey responses (via Qualtrics) could be aggregated manually or displayed through the Qualtrics results, but a new teacher-centric approach could most impact pedagogical decisions for the reflective discussions in class following the EcoMOBILE filed trip. For instance, the instructor could spend the discussion focused on where student content is weakest. Alternatively, the teacher could pair students in heterogeneous groups for better peer learning. For example, one group of students who demonstrated poor abilities in measuring turbidity could be paired with a group of students who demonstrated high proficiency, so that they could learn from each other.
Feature #4 takes the aggregated student performance data (Feature #3) one step further by recommending individualized interventions. On the field trip, this would further centralize knowledge so that the teacher could assist those in need immediately. While a simple “call button” feature within EcoMOBILE (not currently developed) could also assist students who are having trouble with technology or comprehension question, a map feature showing where students are located using the GPS in the mobile devices would be needed to assist the students. Additionally, Feature #4 could extend the benefits of a call button. The call button only helps students who request it; this feature could assist students who do not even know they are not understanding the material by having an instructor explain their errors while they are completing their assignments. A simple progress bar could also help on the field trip so that the instructor could assist students who are significantly behind their peers in progressing through the assignments. Since time is an enormous limiting factor on a field trip, this feature could assist the instructor in catching students up to a the minimal level of exposure so that students are not stuck or left behind.
Moreover, this feature could be extremely useful in class following the field trip by suggesting which students should answer which questions. After clicking on a sub-genre (i.e. defining terminology) or an individual question (i.e. “Which factors affect D.O.?”), Feature #4 could recommend which students should respond (i.e. if James and Peter did not demonstrate an understanding of dissolved oxygen, they could be highlighted to respond to this question in class). This feature could optimize pedagogical decisions like cold calling to maximize student comprehension.
The final suggested feature adapts learning paths to individual student (or group) needs. Feature #5 could tailor homework assignments or review questions based on previous responses so that students spend less time with “busy work” or review questions, and more time acquiring knowledge and know-how where they most need it. For highest achieving students, the system could recommend extension options so they do not feel held back by the tendency to “teach to the middle.” In addition to personalizing learning plans, this feature could also scale tests for a more personalized assessment; if a student already demonstrated proficiency in the stealth assessment, they might not need to be tested on the subject twice. This competency-based approach towards education allows for students who already mastered the material to move on to more challenging subject matter, while simultaneously adapting down to students to practice more until they demonstrate adequate proficiency.
Limitations:
There are technological challenges and theoretical considerations for each of the features in the data back design. It would be necessary to talk with the developers to see whether the current platform supports these features and whether or not the log files could be organized so that student output could easily be centralized. Since ARIS is open-sourced, a tool to sit on top of the platform that could aggregate the data could feasibly accomplish this task, but the researchers should ask at the developer’s conference whether there is capacity for embedded assessment.
Moreover, each subsequent feature in Figure 3, moving downwards, illustrates a greater challenge in implementing the features. More open-ended questions require graders to manually assess the output. Binary rubrics (questions scored as either right/wrong) could be easier to automatically assess, but this limits teachers from custom-point rubrics (i.e. 0-5 points) that differ by each instructor. The features list also holds an assumption that there is significant variability in student responses; if students are progressing at about the same rate and students demonstrate similar levels of proficiency, then tools to intervene with low and high performers do not have as great of a benefit.
Finally, there are numerous research considerations to take into account when deciding the impact of a data back system. Some of the features could impact affective measures or detract from the process. If the purpose of the field trip is to allow students to tinker with scientific devices, learn to fail on their own, have exposure to a STEM career, and reflect holistically on the EcoMUVE simulations, then requiring more student output in order to better aggregate the data could negatively impact student performance. More research is necessary in order to analyze the optimal benefits for teachers while still allowing students to explore an ecosystem and feel engaged while learning.
Conclusions:
A data back design within EcoMOBILE could serve as a powerful stealth assessment that limits survey fatigue, improves student learning, and transforms the role of the teacher. Students should feel engaged in the learning process, and augmented reality systems like EcoMOBILE assist in this goal, but could always be improved. With a data back design, teachers could be empowered to assist students who are most in need, either immediately on the field trip or in the reflective class discussion, while personalizing future assignments in a competency-based model. As researchers continue to evaluate the EcoMOBILE system, features that impact student learning vis-à-vis improved teaching ought to be considered.
Implementing a data back system does not come without technical challenges and theoretical considerations. As such, researchers and developers should discuss their designs with teachers in order to gauge which features would be most useful in a minimal viable product before further iterating on its design. Improving the student and teacher experience while optimizing learning paths is a difficult but noble goal for future versions of this study.
[1] See HGSE T561 student research from Elliot Mandel
[2] See Val Shute on Stealth Assessment
