Language Model Analyst - Fully Remote | Upto $20/hr Part-time

About the Role

The day-to-day responsibilities of this role will include conducting fact-checking using trusted public sources and external tools, generating high-quality human evaluation data, and assessing the reasoning quality, clarity, tone, and completeness of responses. The ideal candidate will have a strong background in domains requiring structured analytical thinking, such as research, policy, analytics, linguistics, or engineering, and will be proficient in Excel and other data analysis tools.

Mercor is committed to fostering a collaborative and dynamic work environment, where team members are encouraged to share their ideas and expertise. As a remote worker, the successful candidate will be expected to be self-motivated, disciplined, and able to work independently, with regular check-ins with the team to ensure seamless communication and coordination.

What You Will Do

• Conduct fact-checking using trusted public sources and external tools to ensure the accuracy of language model responses
• Generate high-quality human evaluation data by identifying response strengths, areas for improvement, and factual inaccuracies
• Assess the reasoning quality, clarity, tone, and completeness of responses to evaluate the performance of language models
• Ensure model responses align with expected conversational behavior and system guidelines
• Collaborate with cross-functional teams to identify areas for improvement and develop strategies to address them
• Use Excel and other data analysis tools to analyze and interpret complex data sets
• Develop and maintain databases to track and store evaluation data
• Create reports and visualizations to communicate findings and recommendations to stakeholders
• Participate in regular team meetings to discuss progress, share knowledge, and align on goals and objectives
• Stay up-to-date with industry trends and developments in AI research and language model evaluation

What We Are Looking For

• Bachelor's degree in a relevant field, such as linguistics, computer science, or engineering
• Native speaker in Tamil, with excellent writing skills in English
• Significant experience using large language models and evaluating their performance
• Strong attention to detail and analytical skills, with the ability to identify and address complex issues
• Experience working with data analysis tools, such as Excel, and programming languages, such as Python
• Background or experience in domains requiring structured analytical thinking, such as research, policy, analytics, or engineering
• Excellent communication and collaboration skills, with the ability to work effectively in a remote team environment
• Strong problem-solving skills, with the ability to think critically and creatively
• Ability to work independently, with minimal supervision, and manage multiple tasks and deadlines

Nice to Have

• Prior experience with RLHF, model evaluation, or data annotation work
• Experience writing or editing high-quality written content, such as technical reports or research papers
• Experience comparing multiple outputs and making fine-grained qualitative judgments
• Familiarity with AI research and development, including machine learning and natural language processing
• Experience working with remote teams and collaborating with stakeholders across different time zones

Benefits and Perks

• Competitive hourly rate, up to $20/hour
• Opportunity to work on cutting-edge AI research and development projects
• Collaborative and dynamic work environment, with regular team meetings and social events
• Flexible working hours, with the ability to work remotely from anywhere in the world
• Access to cutting-edge tools and technologies, including data analysis software and programming languages
• Opportunities for professional development and growth, including training and mentorship programs
• Recognition and rewards for outstanding performance, including bonuses and promotions
• Comprehensive benefits package, including health insurance and retirement savings plans
• Generous paid time off policy, including vacation days, sick leave, and holidays

• To stand out as a candidate, be sure to highlight your experience with large language models and data analysis tools, such as Excel, in your resume and cover letter.
• Practice your fact-checking and data analysis skills before applying, as these will be essential to success in this role.
• Be prepared to provide examples of your work, such as reports or research papers, to demonstrate your writing and analytical skills.
• Research the company and the role thoroughly, and be prepared to ask informed questions during the interview process.
• Consider taking online courses or attending webinars to learn more about AI research and development, and to stay up-to-date with industry trends and developments.
• Be sure to showcase your ability to work independently and manage multiple tasks and deadlines, as these will be critical to success in this remote role.