Youth Mental Health for California Department of Health Care Services
In January of 2023, my team at Vidlet was contracted by the California Department of Health Care Services to lead a research initiative for youth mental and behavioral health. In response to the growing youth mental health crisis, DHCS seeks to introduce no-cost mental health resources to children, teens, and young adults living in California. My team conducted a three-week study with California teens, consisting of digital diaries, two rounds of TestFlight app testing, quantitative surveys, and focus groups in order to make vendor recommendations for DHCS’ new offering.
Objectives
DHCS informed us that they were considering two potential vendors – Kooth and ModernHealth – to support their mental health resource offering. Both vendors currently offer mental health resource platforms and behavioral coaching sessions, which DHCS hoped to test with youth participants. In tandem, designers at McKinsey (hired by DHCS) were working with both Kooth and ModernHealth to reimagine their current platforms as new mobile apps in TestFlight. These new apps would need to be tested by participants and evaluated in comparison to the current platform offerings of each vendor.
Our objective was three-fold:
Evaluate current vendor platforms
Test new vendor platforms
Make vendor recommendations to DHCS based on user insights
Research Methods
Because we were asking participants to evaluate four different solutions (two existing platforms and two new apps), we chose a combination of quantitative surveys, digital diaries, and focus groups.
As participants tested each solution, they documented their experience with digital diaries using Vidlet’s mobile video capture technology. Once they had finished testing that specific solution, they were instructed to complete a quantitative survey evaluating their experience based on metrics surrounding content, usability, design, language, and overall experience. At the conclusion of both phases, existing platform evaluation and new platform testing, participants took part in a virtual focus group session with our research team to discuss their experience more in-depth.
TestFlight Testing
In order to test Kooth and ModernHealth’s updated solutions, we directed participants to download TestFlight and access each app. As they explored each app, participants answered a set of video questions on their mobile phone.
Immediately following, participants completed a quantitative survey in which they evaluated their experience based on the same metrics used for the ‘existing platform’ evaluation.
Behavioral Coaching Sessions
THE PROCESS
In week one of our research, we directed participants to complete two coaching sessions, one with each potential vendor. Immediately after concluding their coaching session, participants completed a survey to evaluate their experience. In addition to this quantitative exit survey, participants also completed a set of mobile video questions to elaborate on their responses.
Participants reviewed each coaching session based on the following metrics:
Ease of connecting with a coach
Quality of session
Length of session
Quality of coach
Relevance of content
OVERVIEW
Challenges
The primary challenge we anticipated was with participant management, specifically with such a young sample (the youngest participant being 12-years-old). We knew participant management would be challenging both from a technical support standpoint, with virtual coaching sessions and TestFlight downloads, as well as with attrition over a three-week period during which participants would have to complete an extensive set of tasks: four (4) platform tests, four (4) quantitative surveys, four (4) digital diaries, and two (2) focus group sessions.
To prepare for these challenges, I compiled an orientation PDF for participants and parents to help them learn what to expect for each week of the study and outlined detailed instructions for each task. Each participants was followed up with a minimum of 4 times per week to ensure engagement and completion of all necessary tasks.
Digital Diaries & Virtual Focus Groups
The qualitative component of our research took two forms: digital diaries (video questions answered via Vidlet’s mobile app) and moderated focus groups via Zoom. The former were completed by participants as they walked through each solution: Kooth’s and ModernHealth’s existing platforms as well as their new apps in TestFlight. The latter were conducted at the conclusion of each phase.
In these sessions, we asked participants open-ended questions about their experience using each solution, diving deeper into their likes and dislikes, and having participants compare and contrast each solution they had interacted with. The purpose of this was to expand on the digital diary and survey findings in a more interactive format.
Participant feedback was organized in Miro to develop insights which would later be shared with the DHCS team.
Recommendations
Our team compiled the results of our surveys, analyzed 6 hours of mobile diary content, and mapped the insights of our focus group sessions on Miro to illustrate our findings. We found that while Modern Health’s existing platform offered a more favorable coaching session experience, Kooth’s updated app in TestFlight was the participant favorite. However, participants highlighted certain features in the new Modern Health app that they would like to see in a final solution.
We learned that participants saw great value not only in coaching sessions, but in self-paced activities and relaxation-focused exercises (e.g., breathing skills and noise cleanses). Participants responded positively to gamification, and they placed a very high value on an aesthetically pleasing interface. While DHCS had hypothesized that end users may feel more comfortable engaging with a behavioral coach via text-based chat for reasons of timidity or a desire for privacy, we found the opposite to be true – participants sought a high level of personalization and highly praised the ability to connect face-to-face with a coach.
We delivered our final recommendations in a read out with the DHCS team and McKinsey design team, both for the final vendor selections and for specific design features to be implemented in the final app. DHCS is currently in the processing of finalizing vendor agreements to implement these new mental health resources at no cost to California youth.