The logistics and costs of qualitative research can be overwhelming, especially for fast-turn and global projects. The promise of “AI Moderation” is to get hundreds of responses across the globe, at the convenience of respondents’ schedules, in a matter of hours or days. At Vital Findings, we’ve partnered with a number of different AI “Moderation” platforms and tools, and we’re ready to bring our initial findings to you. Here we give you the answers to these questions:
- What is AI “Moderation”?
- What are the pros and cons?
- When should you consider using it?
What is AI “Moderation”
In an AI “moderated” interview, you program your discussion guide, and allow the AI to probe each question with 1-3 dynamic follow-up questions. Respondents can provide an answer using voice or text. Furthermore, quantitative, closed-ended questions can be included, providing a mix of quant and qual. The promise is fast qual at scale—without a human moderator, we can interview hundreds of respondents in days or even hours, on a global scale.
Here are some examples of how we’ve used this technology:
- We ran a TV ad test for a social media company. Each question allowed for one dynamic AI follow-up. The follow-up was different for every respondent. For instance, the AI “moderator” asked one participant to clarify their meaning, another to explain why the message was important to them, and another to share why they thought the advertised product wasn’t for them.
- We ran a B2B study with insurance brokers. In this case, we offered the option of a human moderator or an AI “moderator.” Brokers who chose an AI “moderator” liked that they could click to do the interview immediately vs. scheduling with a human moderator, and were pleasantly surprised by how nimble the AI “moderator” was.
- We ran a brand image study for a sports drink brand that was looking to extend its reach to a broader audience. The goal of this use of AI was to take findings from a series of in-person focus groups and see how well they scaled to a broader group of consumers. The findings gave us confidence that the focus group findings held up at scale, and also gave us a larger sample size that we could slice by key demographic groups.
What we like about AI “moderated” interviews:
- Voice Responses: We love the option to speak responses instead of typing, and the AI quickly understands and transcribes them.
- Flexible Questioning: An AI “Moderator” can act in unexpected (good) ways. For example, in the social media ad test, when a respondent hesitated and stammered when asked a question, the AI pivoted to asking about what in the ad seemed confusing or unclear.
- Flexible Timing: Interviews can be conducted anytime without worrying about moderator availability.
- Efficiency: We can complete hundreds of short interviews in a few days.
- Advanced Tools: Impressive analysis tools. For example, sentiment analysis isn’t a typical output, but we can ask the AI to perform sentiment analysis on the fly.
- Positive Respondent Feedback: While the experience benefits from novelty, respondents find it engaging and enjoyable.
Challenges we see with AI “moderated” interviews:
- Limited Nimbleness: We purposefully keep “moderation” in quotes when discussing AI interviews. AI can’t turn on a dime. It can’t fully follow a respondent’s train of thought if they jump around the guide. It can’t respond to nonverbal cues. It’s not thinking about previous interviews and listening for themes during the interview. It’s limited by the context it’s given.
- Paraphrasing Issues: AI often paraphrases responses to demonstrate listening (e.g., “Comfort is certainly important…”). While this helps respondents feel like the AI is listening, it could bias some types of research.
- Probing Repetition: AI may repetitively probe without knowing when to stop.
- Inconsistent Question Phrasing: AI may phrase questions differently for each respondent, and researchers may not always have visibility into these variations.
- Resistance from Traditional Survey Panels: Panel members often prefer surveys over AI interviews, so respondent pools need to be selected wisely.
- Quality Control Concerns: Many AI startups rely on panels to prevent fraud and bots, which isn’t enough quality control for us.
What Does AI “Moderation” Work Best For?
- Speed and Volume: Ideal when rapid, large-scale data collection is essential.
- Evaluative Studies: More suitable for evaluative rather than exploratory research.
- Early Research Phases: Useful in early stages of multi-phase projects to guide questionnaire development or discussion guides.
- Post-Human Moderation: Effective for testing findings on a larger scale after human-moderated research.
- Sensitive Topics: Some respondents prefer talking to AI vs. a human moderator for sensitive topics.
- Busy B2B Professionals: Convenient for hard-to-reach professionals who can take the AI interview immediately after receiving the invitation without having to wait to schedule an interview.
- Later Stage Creative Testing: For late-stage creative testing, AI is great for disaster checks, or to make minor tweaks to creative.
This technology is still in its infancy, and it shows a great deal of promise. There is a lot of momentum behind it—one of the companies in this space won Greenbook’s IIEX Insight Innovation competition this year. However, currently, there are no less than 25 companies in this space or developing solutions for it, and most have been operating for less than a year. We’ve found that they vary considerably in capabilities and stages of development. That said, we highly recommend experimenting with the technology, and we’re here to help you figure out if it will work for your business challenge, and how to put your data into action.
