Naomi-AI: AI-Driven Guidance Counselor

3. Naomi’s Usability Testing 1

3a. Objective

Our goal is to improve the user and learning experience of 24/7 Teach's learners by evaluating Naomi's ability to meet their specific needs with and without initial guidance from 24/7 Teach. I will also monitor how students use Naomi on a daily basis to answer academic, social-emotional, and career-related queries.

3b. Methodology

Conducted usability testing with 9 high school students to evaluate Naomi-AI’s response quality and user experience.

• Participants: 9 students enrolled in a 24/7 Teach program

• Duration: 1 week

• Method:

  • Phase 1 (Unassisted): Moderated session where students asked Naomi questions without guidance

  • Phase 2 (Assisted): Week-long independent use with prompt guidance

• Focus Areas:
  Academic subjects, social-emotional support, college counseling, and career guidance

Prompt Guidance (Training Intervention)

Provided users with simple strategies to improve AI interactions:

• Be clear and specific

• Provide context

• Break down complex questions

• Use examples and follow-ups

3c. Metrics

How High School Students Rated Naomi’s Responses for Math, Science, History, and English Questions on a Scale from 1-10

Metrics: The metrics above show how the high school participants scored Naomi on a scale from 1 to 10 for Math, Science, History, and English questions. The blue charts represent the data from the first survey when the students did not receive AI training before using Naomi, while the purple charts represent the second survey completed at the end of the week with AI training. 

Observation: Naomi received better scores with homework assistance in humanities subjects than in Science, Technology, Engineering, and Mathematics (STEM) subjects. However, the ranking for History slightly improved after receiving AI training, while the score for English declined narrowly. I concluded that Naomi perform best with fact-based humanities like History.

The percentage of participants who would refer Naomi to a friend

In conclusion, as shown above, a majority of the high school participants would use Naomi every day for homework, social-emotional, college, and career counseling. Additionally, they would refer Naomi to a friend.

Successes

User Testimonies

• “The steps to finding the solution to the question are clear, concise, and accurate. Naomi even tells you the appropriate formula that you should use, and the steps are numbered.“

• “I love this answer, especially the first guide about affirming friendship. As to social-emotional response, it provides deep understanding and empathy.“

• “10 [out of 10], a really solid response that doesn't just summarize the events in the novel, but also offers questions to the user to think about while examining the events and characters in conjunction to the main theme of the novel.”

• “It gave a thoughtful answer that accounted for where one lived and the industry in which they worked, but also that there are more important things to consider, like job stability, work-life balance, and fulfillment.”

According to the above testimonials, Naomi successfully provides clear, concise solutions to math and science problems with step-by-step formulas. Naomi can put herself in the users' point of view and help them summarize literary sources and historical events. Lastly, Naomi empathizes with the users when they ask for guidance on nonacademic subjects, including interpersonal communications and career mobility.

Areas for Improvement

User Feedback

• “It could be better if the response was structured and numbered/bulleted, rather than feeding the user a paragraph for an answer.“

• “I think this AI can improve if they AI actually did the problems with us, and maybe provide a final answer.“

• “It would a lot more efficient if Naomi incorporated working links to the resources that it names for the user, this could speed up user's experience, and make navigating to resources a lot easier.”

• “ [diagram]: provide diagrams when the user asks for them because the guidance might not be helpful for someone who is a visual learner.”

Although the participants would refer Naomi to their peers, Naomi needs improvements in the following areas:

• Problem 1: Text Blocks: Long Naomi's responses are presented in a block format without any spacing, which can cause cognitive overload for the reader.

  • Actionable Plan 1: Improve Readability: Program Naomi to use bullet points and numbering to break up responses for users.

• Problem 2: Nonfunctioning Links: Naomi does not provide functioning links and assigns resources for users to search for themselves.

• Actionable Plan 2: Expand Responses: Include links to educational resources and material directly into Naomi’s responses.

• Problem 3: Incomplete/Incorrect Answers: Naomi either does not provide the final answer or misinterprets literary passages for open-ended questions.

• Actionable Plan 3: Train Naomi Further: Prepare Naomi to answer accurately in simple language. Measure the performance of Naomi’s answers.

• Problem 4: No Diagrams/Learning Graphics: Naomi does not generate diagrams or other learning visuals with her responses.

• Actionable Plan 4: Enhance Naomi’s Functionality: Program Naomi to use complex algorithms and create visuals.

Lessons

I learned that the best way to observe the feasibility of your product is to perform a usability test with target clients without training materials first as the control. Then, provide training materials to the participants as an experimental variable, reiterate usability testing, and analyze the shift in opinions and behaviors after receiving the experimental variable. Lastly, I learned that having similar question formats between the initial and final usability testing ensures consistency in data visualizations between the two usability tests and feasible data analysis.

4. Naomi’s Usability Testing 2

4a. Objective

I aimed to evaluate and showcase the significant improvements made to Naomi-AI since Usability Test 1, demonstrating our commitment to enhancing its usability and user experience. Additionally, I wanted to evaluate the effectiveness of Naomi-AI’s instructional guides, which the instructional design team and I prepared.

4b. Methodology

I devised Usability Test 2 by recruiting 9 high schoolers from the previous usability test and asking them to complete a short remote synchronous (moderated) test over Zoom.

Target Users: 9 High Schoolers enrolled in a 24/7 Teach program

Duration: 1 Day

Tasks: We gave the 9 students instructional guides to complete the 2-hour synchronous (moderated) remote usability test via Zoom. The students provide feedback on their experience with the updated Naomi-AI by completing a short survey.

4c. Metrics

Students’ Rating the Overall Experience and Helpfulness of Naomi-AI during Usability Test 2

During Usability Test 2, 9 high school students rated the overall user experience of the upgraded Naomi-AI as 8.56 out of 10. Additionally, Naomi received an overall helpfulness score of 8.89 out of 10.

Successes