We are looking for experienced Subject Matter Experts (SMEs) to support an AI conversation evaluation project.
The role involves reviewing AI conversations involving real-world interpersonal and everyday decision-making scenarios, identifying missing context, and providing structured expert feedback to improve conversational quality.
What You’ll Do
• Review AI-generated conversations and recommendations
• Identify important context that may have been missed
• Suggest relevant follow-up questions that could improve the quality of recommendations
• Assess whether additional information could affect an AI recommendation
• Distinguish important information from non-essential details
• Conduct structured qualitative reviews and document findings
• Collaborate with research teams to maintain consistent evaluation standards 108815 - Text Situated Alignm…
Who We're Looking For
Professionals with 3-5 years of direct relevant experience in one or more of the following areas:
• Social Work - MSW and relevant professional license/credential
• Counseling / Psychology - relevant graduate qualification, PhD/PsyD, or applicable professional credential
• Qualitative Research / HCI / Communication
• Anthropology / Sociology
• Mediation / Coaching
• Student Services
• Organizational Consulting or related advisory fields 108815 - Text Situated Alignm…
Preferred Experience
• Client or participant interviewing
• Qualitative research or case review
• Conversation analysis or qualitative coding
• Experience gathering information when initial context is incomplete
• Ability to identify information that is critical to a recommendation
• Experience reviewing consultations, interviews, coaching sessions, or similar interactions