Remote | Marathi-English AI Safety Red Team Evaluator — $20–$30/hour

About the Role

Day-to-day, you will work on structured adversarial scenarios to stress-test conversational AI models and agents. Your attention to detail and ability to think adversarially will be crucial in identifying vulnerabilities and classifying risks. You will also collaborate with the team to document findings, produce clear reports, and maintain accuracy and consistency across evaluations.

The role is part-time and flexible, with opportunities to work on a range of projects focused on AI safety evaluation, bilingual red team testing, and conversational model assessment. You will have the chance to apply your skills to real-world problems, working with a team of experienced professionals who are passionate about AI safety.

What You Will Do

• Test AI systems using structured adversarial scenarios to identify safety weaknesses and classify risks
• Review English and Marathi AI outputs for safety, reliability, bias, misinformation, and harmful-behavior risks
• Evaluate model behavior across multi-turn conversations, sensitive topics, and edge-case prompts
• Identify vulnerabilities that require stronger safety controls, clearer refusals, or improved response quality
• Annotate failures, classify vulnerabilities, and flag recurring safety patterns
• Apply taxonomies, benchmarks, and project-specific playbooks to keep testing consistent
• Assess misuse cases, bias exploitation, prompt-injection scenarios, and socio-technical risk patterns
• Generate high-quality human evaluation data through careful review and structured judgment
• Produce clear reports, datasets, test cases, and written summaries that support model improvement
• Document findings reproducibly so results can be reviewed, compared, and acted upon
• Explain risks clearly for both technical and non-technical audiences

What We Are Looking For

• Native-level fluency in both English and Marathi
• Prior experience in AI red teaming, adversarial testing, cybersecurity, trust and safety, or conversational AI evaluation
• Ability to think adversarially while staying structured, careful, and methodical
• Experience using frameworks, benchmarks, or rubrics rather than unstructured testing alone
• Strong written communication skills and ability to explain safety findings clearly
• Comfort reviewing text-based content involving sensitive topics under clear guidelines
• Adaptability across project types, safety categories, and evaluation workflows
• Formal degree in AI safety, cybersecurity, linguistics, policy, trust and safety, social science, psychology, writing, data evaluation, or technical analysis
• Practical experience in red team testing, model evaluation, content risk analysis, or structured review work

Nice to Have

• Experience with adversarial ML concepts, jailbreak datasets, prompt injection, RLHF/DPO attack patterns, or model behavior testing
• Cybersecurity experience such as penetration testing, exploit analysis, reverse engineering, or security assessment
• Socio-technical risk experience involving harassment, misinformation, abuse analysis, bias testing, or conversational AI safety
• Creative probing background, including psychology, acting, writing, role-play design, or unconventional adversarial thinking

Benefits and Perks

• Competitive hourly rate of $20-$30
• Flexible, part-time work arrangement
• Opportunity to work on a range of projects focused on AI safety evaluation and bilingual red team testing
• Collaborative team environment with experienced professionals
• Professional development opportunities in AI safety and red team testing
• Access to cutting-edge tools and technologies
• Remote work stipend and equipment allowance
• Health and wellness benefits, including mental health support
• Generous PTO and holiday policy

• Develop a strong understanding of AI safety evaluation and red team testing principles to stand out in your application.
• Showcase your experience with conversational AI models and socio-technical risk review in your portfolio or resume.
• Be prepared to discuss your approach to adversarial testing and vulnerability classification during the interview process.
• Highlight your ability to communicate complex safety findings clearly and effectively to both technical and non-technical audiences.
• Research the company's approach to AI safety and red team testing to demonstrate your knowledge and enthusiasm for the role.
• Consider taking courses or attending webinars on AI safety, cybersecurity, and conversational AI to enhance your skills and stay up-to-date with industry developments.