Behind the Build: Designing Post-Quiz Review

If you’ve always considered yourself a “bad test taker,” there’s a new Codecademy feature that’ll help you feel more confident taking a quiz. Introducing post-quiz review! Utilizing delayed feedback, post-quiz review helps you understand mistakes through answer explanations to retain key information for longer. 

As someone who loves to learn, I’m excited to introduce myself and this feature! I’m Kat Minor, a Codecademy learner and an Associate Product Designer currently working on the TA Labs team. Our team is responsible for creating the learning environment, the interactive space our learners use to learn new skills, upskill their abilities, and gain hands-on experience with concepts.   

As the design lead for this project, I was tasked with giving learners the opportunity to reflect on both their strengths and knowledge gaps after taking a quiz. By focusing on filling knowledge gaps, we can help our learners be better prepared for the next time that they encounter a similar question or concept. Through encouraging our learners to reflect on their answers, we were able to boost average assessment scores and increase time spent engaging meaningfully with our course material. 

But designing a feature that truly enhances the learning experience requires more than just hope to help our learners. I needed to make sure that I understood the learning science behind what would make this feature successful. 

We collaborate closely with our Curriculum team when determining what changes we should make to the learning environment. For this particular project, Alex DiStasi from our Curriculum team pitched and drove the strategy. Their insights into delayed and immediate feedback guided the feature’s creation. 

Both types of corrective feedback are shown to help with information retention but are slightly different from one another. Understanding this concept as the designer was important to ensuring that learners received the proven benefits of feedback. 

Immediate feedback is just what it sounds like, receiving a response right after you perform an action. We already offered this on quizzes through brief popup text explaining why your answer is correct or incorrect immediately after you answer. Here are some more examples of immediate feedback: 

• Seeing a confetti explosion after you get the correct answer. 
• Hearing a little jingle after you open a chest in “The Legend of Zelda.” 
• Feeling your phone buzz when you get the pass code incorrect. 

Delayed feedback is similar to immediate feedback, but instead of happening instantly, it occurs after you complete an action in its entirety. This means delayed feedback can occur 10 seconds, 8 hours, or even a week after the initial action. Here are some examples of delayed feedback: 

• Receiving your test grade with comments after a few weeks. 
• Getting your accuracy percentage after a game of laser tag. 
• Hearing comments about your cooking after submitting it to a baking competition. 

The project: Implement delayed feedback on our quizzes to help learners retain information.   

Both delayed and immediate feedback offer unique benefits to learners. While immediate feedback helps to reinforce understanding in the moment, delayed feedback allows learners to reflect and assess their own knowledge. Since Codecademy quizzes previously only offered immediate feedback, we saw an opportunity to enhance the learning experience by offering the best of both worlds. 

With just over 400,000 quizzes taken every single month (that’s one quiz every six seconds, on average!), we saw updating quizzes to include delayed feedback as an incredible opportunity to have a large, positive impact on our learners. 

Investigation and roadmapping  

We recognized that while learners needed immediate feedback in quizzes to be short to maintain their momentum, a page with delayed feedback allowed for longer, more information-dense explanations. These explanations clarify why certain answers are correct and provide detailed topic overviews, boosting learners’ understanding of core concepts. 

At Codecademy, our Product team operates on an eight-week cycle where we spend six weeks moving full steam ahead on projects and two planning our next move, remediating bugs, and documenting our decision making. This process helps give us enough time to make exciting features for our learners while taking the time to understand if these features are as impactful on learning as we want them to be. 

Implementation 

Due to the vast number of courses and quizzes on Codecademy, we knew that we couldn’t write out specialized explanations for each quiz question all within our six-week cycle. So, we leveraged AI to generate explanations for learners immediately after completing their quiz. We saw an 84.1% positive feedback rating from our built-in feedback system — our explanations got a passing grade from our learners! 

Because our quizzes can get long (some have more than 20 questions!), we wanted to make it easy for learners to find explanations they find most relevant to their learning experience. We figured that most learners would focus on the questions that they got wrong, so we added in-page navigation to allow for learners to quickly find and read further explanations.  

Fun fact: The Product Design team uses tools like Figma to create our designs and prototypes, and we test them using platforms like UserTesting. During this project, I conducted usability testing with the help of our UX Researcher, Sil Lavers, to gather feedback on the feature. Some of my design hypotheses were validated, such as our learners like to see their overall score followed by in-depth explanations. However, one surprising insight was that our learners found it useful to have explanations for both correct and incorrect answers. These discoveries helped me structure the page to help learners find information most applicable to them. 

Using the insights we gained from the usability testing, we updated the quiz summary to provide learners with a birds-eye view, enabling them to see their overall score, how many questions they got correct and incorrect, and the ability to retake the quiz, all at a glance. We recommend learners earn a certain minimum score before progressing, so having the opportunity to quickly review and retake a quiz streamlines this process.  

Following this “overall score” section, we included more in-depth explanations of each question, as well as a thumbs up/down rating system so learners could tell us how useful they found each one. Because these explanations were generated by AI, we included the rating system as a way to prevent bad explanations from slipping through.  

Ship 

Following the launch of this feature, we saw clear signals that learners were engaging with and enjoying the new post-quiz review feature. We saw a bump in time spent on quizzes, from 4.13 minutes to 4.26 minutes spent on average per quiz. This indicates that people are utilizing this post-quiz review to better understand concepts in the quiz that may have initially left them puzzled.  

In addition, average assessment scores are up from 84% to 87%! That’s a positive signal that our post-quiz review makes it easier for our learners to achieve higher scores. Learners can even return to their quiz review, should they ever need to jog their memory a bit. 

Retrospective 

It’s been less than two years since the launch of ChatGPT, a tool that, in my opinion, changed the learning landscape. Learning with AI still feels like a whole new world, there are so many opportunities to make online education more accessible, personalized, and engaging. The positive feedback from our learners regarding post-quiz review motivates me to continue to pitch features whichd can help push the Codecademy experience even further. 

This feature expands the tailored learning experience that our AI-powered code-explain and error-explain tools offer learners. Post-quiz review gives learners more power to fill their knowledge gaps, allowing us to customize each learner’s experience through prioritizing areas where they have more opportunities to grow. 

As a designer at Codecademy, I find it incredibly exciting and rewarding to shape the learning experience for so many. Understanding our learners, helping them to achieve their own goals, and continuously discovering ways to create a more engaging experience makes me look forward to a future where people who learned with Codecademy go on to build incredible things. 

Snaps 

Alex DiStasi – Alex truly helped me understand the learning principles behind this feature. As the Instructional Designer for our team and the one responsible for writing the pitch for this project, she really deserves a shout out for making this whole project happen. It was so interesting to read more into the research that she provided; she had a huge impact on how I thought about post-quiz review. 

Mark Hannallah – Mark is an amazing Product Manager; it was incredible to work with him on post quiz review. My favorite aspect about working with him is that he keeps the team motivated, he grabbed highlight clips of our user testers being excited about the post-quiz review feature and shared it at our next meeting. I love the idea; I can’t wait to steal it from him for my next project. 

Irene Robb and Nar Shahin – Working with Irene and Nar was incredible. I really appreciated the work that they put into checking the accessibility of the project. They taught me more about how screen readers work, which was great information to bring into my next project. Both were also incredibly dedicated to bug bashing, especially making sure what was in production matched the Figma file. 

Jerimie Lee – Jerimie is my design mentor at Codecademy and brainstormed with me throughout the project. He had a lot of out-of-the-box layout ideas that helped me think further on how a learner would actually utilize our feature. I learnt a lot from his user-centric approach. 

Sil Lavers – Thank you to Sil for guiding me through the research process at Codecademy. She taught me how to do effective remote usability testing that helped to shape the project and synthesize the results. It was wonderful to have her support as this was my first project at Codecademy. 

Emily Lee – Emily is a great source for understanding our learners. She works as a UX Writer but her work and advice often move beyond just the surface text. I really appreciated her feedback, the advice she gave was logical and made me look at the at the project in new ways.

The post Behind the Build: Designing Post-Quiz Review appeared first on Codecademy Blog.