An augmented reality educational storytelling app.
August 2020 -
My master's project combined what I learned from my previous coursework at Georgia Tech, as well as my long-rooted passion in education and storytelling. With the emergence of augmented reality (AR) within consumer technology, I wanted to use this technology to teach STEAM topics. My partner and I sought to harness the power of cameras and LiDAR sensors on iPad to add context and visualization to STEAM education.
How might we improve learning STEAM concepts in schools through the use of Augmented Reality?
The STEAM Playground is an AR experience for iPad that teaches middle school students mathematical concepts within the context of a story. After narrowing down concepts that students struggle with, we decided to cover fractions with a farmer managing his land. Using laser cut patterned pieces and our iPad app for character recognition, students were able to place the farmer's crops and solve problems about fraction equivalence.
In order to obtain information about our topic, we collected information using the following research techniques:
We conducted research by checking the Georgia Standard Department of Education (DOE) curriculum for elementary school students, uncovering topics that students struggle with and corresponding class activities.
We also searched for online sites and games that teachers use, such as Mathisfun.com and Mathgames.com.
From our primary research, we discovered that math concepts in between 4th grades are largely abstract and disconnected from concrete concepts. Students have trouble visualizing concepts in math class.
To delve further, we sought to obtain data from elementary school teachers. My team developed a Qualtrics survey to understand the techniques teachers use as well as their pain points. We also sought to collect contact information for future research.
Directly contacting the Georgia DOE, we collected 12 teacher responses. We uncovered three major takeaways:
4th grade students struggle with fraction operations, word problems, and applying math to reality with visualization.
Delving into interviews, we presented teachers the same online survey questions but encouraged them to go in-depth. This was to validate our survey responses as well as determine which activities were helpful.
Recruiting 7 math teachers and specialists around Georgia, we learned that when students learn with tangibles, such as folding paper or coloring, it helps solidify concepts. Students begin with the concrete math, then moving towards the representational and abstract. (CRA)
Letting students find solutions on their own helps gain a deeper understanding and apply knowledge.
Taking these insights into account, my team created an Affinity Map to group ideas together. These three core themes emerged.
From our three core themes, we developed a list of relevant Design Implications.
1. The solution should help students practice fractions in a way that facilitates recognition and recall.
2. The solution should lead to a better understanding of what a fraction is and how it is represented.
3. The solution should use real problems in context for students to better understand not only HOW, but WHY.
4. Solution should include gamifications and demonstrations to encourage students.
5. System should foster student’s reasoning, curiosity and problem solving skills.
From our design implications, we updated the problem statement.
How might we create a tool that connects concrete and abstract learning, reinforce fraction foundations with visuals, manipulatives, and context, while also grows students' curiosity?
After updating our problem statement, we develooped three key themes for our solution.
This teaching tool helps prepare teachers before lessons.
Extension of in-class activities helps reinforce students' understanding and foundation.
Fraction learning tools can cover content from the Georgia DOE curriculum.
After talking with several Georgia DOE experts, we observed several class activities from the Georgia Standard curriculum. We chose a class activity called Pattern Block puzzles, in which students arrange plastic geometric shapes to show fraction equivalence.
The core storytelling interface introduces lessons and stories for students to solve based on the Georgia Standard curriculum.
In our case, we introduced the story of a farmer trying to decide which crops to grow on his farm.
Students are shown similar problems at first to understand the concept, then are able to explore their own solutions in Augmented Reality.
We developed an interface in Unity that uses a camera to display assets over an image target. Using Optical Character Recognition (OCR) This lets the laptop detect patterns on the worksheet made with the color tiles.
Our Unity interface scans for patterns and projects relevant 3D models over the student worksheet, as well as a simulated farmhouse with sounds to feel more immersive.
If an answer is correct, the student is rewarded with confetti and a congratulations. If it is incorrect, the student is then encouraged to try again.
Laser cut wooden tiles are the key to filling out the worksheet.
As students work through the story activities, they arrange the tiles in a proper order on the designated worksheet tile and scan with the iPad.
Different outcomes are accounted for, and tiles that don't fit within the confines of the hexagon are counted as errors.
Feedback is recorded on a pre-made paper worksheet. There are spaces to record answers, as well as a specific place to act as a backdrop for the AR features.
Having created our basic prototype, we sought to evaluate it with our intended users. Reaching out to parents and teachers around Georgia, we scheduled several students for evaluation.
Determine if the interface works with usability heuristics with middle school students.
We had students walk through the complete story experience by guiding us remotely through four key tasks.
1. Navigating through the iPad interface
2. Arranging the pattern tiles
3. Scanning the tiles to check answers
4. Recording findings on the worksheet.
Afterwards, experts assign ratings according to Nielsen's 10 Usability Heuristics and complete a System Usability Scale (SUS) questionnaire.
Our feedback was then factored in and added to the final iteration after several design changes.
After consulting with the team from our evaluation with experts, we corroborated the scores from our usability testing and comments. Our researchers then distilled the data into insights that led to a final revised iteration of the prototype.
In order to close the disconnect between the iPad Figma prototype and the laptop Unity interface, we ported our Unity project over to XCode as a native iPad app.
This reduced confusion by keeping both interfaces on one singular device.
To reduce the number of paper materials needed, we added input sections on the app interface. This accounts for a future state in which teachers can easily grade student work.
The paper acting as a backdrop can then be reused for work for a variety of class activities instead of a single-use.
From the STEAM playground, it was a culmination of my work at Georgia Tech. It was incredible seeing how this solution could impact learning for students using tangibles paired with AR. I wish to pursue this further with more iPad learning experiences in the future.
In conclusion, this project was an incredible experience to work on, and I would like to thank the Georgia DOE for working with us in the Georgia Tech MS-HCI program.