DevOps Engineer - AI Model Evaluator

About the Role

The day-to-day responsibilities of this role are both challenging and rewarding, requiring a deep understanding of cloud platforms, Kubernetes, CI/CD systems, and infrastructure automation. The engineer will be tasked with identifying bugs, edge cases, and failure modes in model outputs, comparing strengths and weaknesses of different models, and collaborating with the team to apply these insights to real-world infrastructure engineering scenarios. Given the nature of the work, the ability to work independently and as part of a distributed team is essential, with strong communication skills being a key asset.

Mercor's commitment to innovation and excellence provides a dynamic work environment that encourages professional growth and learning. The company's investors, including Benchmark, General Catalyst, Peter Thiel, Adam D'Angelo, Larry Summers, and Jack Dorsey, underscore its potential for making significant impacts in the tech and AI sectors. For professionals looking to contribute to the forefront of AI and digital technology, this role offers a unique chance to be part of a community that is shaping the future of these fields.

What You Will Do

• Use frontier AI coding agents to complete and evaluate complex infrastructure engineering tasks.
• Review model-generated implementations involving cloud platforms, Kubernetes, CI/CD systems, and infrastructure automation.
• Identify bugs, edge cases, reliability issues, and failure modes in model outputs.
• Compare outputs from multiple frontier models to assess strengths and weaknesses.
• Apply professional engineering judgment to realistic infrastructure engineering scenarios.
• Collaborate with the team to integrate insights from model evaluations into infrastructure design and development.
• Develop and maintain scripts and tools to automate the evaluation and comparison of model outputs.
• Participate in the refinement of the AI model evaluation process, suggesting improvements based on findings and industry trends.
• Engage in knowledge sharing and best practice development across the team and with external partners.
• Ensure compliance with industry standards and best practices for cloud security, compliance, and operational efficiency.

What We Are Looking For

• 2+ years of professional DevOps, SRE, or Cloud Engineering experience.
• Experience with AWS, Azure, GCP, Kubernetes, Terraform, CI/CD pipelines, or observability tooling.
• Regular use of AI coding agents like Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools.
• Ability to evaluate model-generated infrastructure and reliability engineering solutions.
• Strong understanding of cloud computing platforms and their applications.
• Experience with automation and scripting languages (e.g., Python, Bash).
• Ability to work in a fast-paced, dynamic environment with a focus on innovation and problem-solving.
• Excellent communication and teamwork skills.
• A bachelor's degree in Computer Science, Engineering, or a related field.

Nice to Have

• Experience supporting production-scale systems, preferably in a cloud environment.
• Knowledge of machine learning principles and their application in infrastructure engineering.
• Familiarity with agile development methodologies and version control systems (e.g., Git).
• Participation in open-source projects or personal projects that demonstrate coding proficiency and innovation.
• Certifications in cloud computing, DevOps, or related fields.

Benefits and Perks

• Competitive compensation tied to accepted work, with a rate of $400 per accepted task.
• The opportunity to work with leading AI research labs and contribute to cutting-edge projects.
• Flexible, remote work arrangements that allow for a healthy work-life balance.
• Access to the latest technologies and tools in AI and cloud computing.
• Professional development opportunities, including training and support for continuing education.
• Collaborative, dynamic work environment with a team of talented professionals.
• Regular feedback and performance evaluations to support career growth and development.

• Tailor Your Resume and Cover Letter: Highlight your experience with DevOps, SRE, or Cloud Engineering, and specifically mention your work with AI coding agents and model evaluations.
• Prepare for Technical Interviews: Review the basics of cloud computing, Kubernetes, and CI/CD pipelines, and be ready to discuss your experience with these technologies.
• Showcase Your Problem-Solving Skills: In your application and during interviews, demonstrate your ability to analyze complex systems, identify issues, and propose solutions.
• Be Ready to Talk About AI and Automation: Show enthusiasm and knowledge about the latest trends in AI and automation, and how you see these technologies evolving in the field of DevOps and cloud engineering.
• Demonstrate Your Ability to Work Remotely: Highlight any experience you have with remote work, and emphasize your self-motivation, discipline, and excellent communication skills.
• Research the Company and Its Investors: Understand mercor's mission, values, and the impact of its investors on the company's vision and goals, to show your genuine interest in the role and the company.
• Negotiate Your Rate Based on Experience: If offered the position, negotiate your compensation rate based on your experience and the value you can bring to the company, considering the $400 per accepted task as a baseline.