User Recall Analysis
Drove an end-to-end data solution to unravel user recall issues within Meta's A/B Testing product and conducted AB Test experimentation to measure success.
The project significantly increased platform activeness that led to growth in incremental revenue. The identified pain point theme has been incorporated into the design best practices across the Ads Org. The work also attracted interest in expanding the analytics team to scale the Qual & Quan partnership model.
What is 'User Recall'
User recall becomes a usability issue when users have to rely heavily on memory retrieval to complete a task, rather than having appropriate cues to guide their actions.
​
Meta A/B Testing Product Context
User Journey
Result Reporting
Test Creation
Entry Point of Test Creation
- Convergence Guidance
- Convergence Edit
- ToolBar
- Duplication
Problem Identification
Users opt into A/B Testing through a toggle
Users complete an ads setup and then publish the Ad
The A/B Testing Bottom Sheet pops up
Convergence Guidance Flow
User Recall Pain Point: users have difficulty remembering what they signed up for after completing the ads publishing task.
Hypothesis: This could result in incompletion of A/B Testing tasks.
Validation
1. Transform the user pain point into data questions.
​
2. Benchmarking & Segmentation.
​
Key Findings:
- The bounce-off rate at the Convergence Guidance entry point was stat-significantly higher compared to other entry points, leading to a lower overall task completion rate.
- This pattern was consistent across various segments of advertiser expertise.
- Users who entered through the Convergence Guidance did not exhibit stat-significant differences compared to those using two of the other three entry points.
​
​
Prioritization
1. Opportunity Sizing.
2. Team alignment.
​
Key Actions:
- [Apr 22] Posted a recall analysis note and led a result readout session.
- [May 22] Restated the project opportunity size during roadmapping session and supported the product review.
- [June 22] Roadmapped!
​
​
​
​
​
​
Solution:
​
1. Provided content updates on the welcome page to remind users that they opted into the A/B Test.
2. Added thumbnails with ad previews to help with users' information retrieval.
Measurement
AB Test Experimentation to measure casual inference.
​
- ✅ [Team Top-line] Stat-sig increase in Active Advertiser.
- ✅ [Team Top-line] Positive signal in incremental revenue.
- ✅ Stat-sig increase in Test Creation.
- ✅ Stat-sig increase in Task Completion Rate.
​
Impact
Goes beyond the project
Built trust
Two additional recall projects have been included in the team's roadmap
Established design best practices
Influenced one project from a different team to address recall issues in the roadmap
Generated interest in scaling the Qual & Quant partnership model
UXR
“Cynthia has been especially good at helping the team articulate good user experience research questions. For example, her investigation into the usage rates of A/B testing through different testing flows (i.e., the 'red route' of A/B testing) has already had an impact on the kinds of work A/B testing is planning to pursue next half (e.g., fixing user recall and addressing retention). Her work also took qualitative research findings and backed them up with solid quantitative evidence which together made the problem clear and measurable.
I very much look forward to working with Cynthia as we combine forces to develop and explore important UX questions."
Senior Design Manager
"Cynthia has proven herself an invaluable member of the A/B Testing
team all up and a core part of our Design team.
​
- Her analyses of product experience opportunities has been instrumental in making prioritization decisions both during roadmapping and over the course of the half.
- Her partnership with PD and UXR on their Recall work is some of the best I've seen at Meta.
- This work led to the prioritization of their recommended solution on our roadmap, which ended up being the most impactful work this half on our team iRev Topline. "