DevOps Engineer - AI Model Evaluator
WFA Digital Insight
The demand for skilled DevOps engineers with expertise in AI model evaluation is on the rise, with the global DevOps market projected to grow 24% by 2027. mercor, a company at the forefront of AI research, is seeking a talented DevOps engineer to join their team. As a DevOps engineer in this role, you will have the opportunity to work with cutting-edge AI technology and collaborate with a team of elite creative and technical professionals. With the remote work trend on the increase, this role offers the flexibility to work from anywhere, making it an attractive option for those looking for a better work-life balance. Before applying, candidates should be aware that they will need to have a strong foundation in cloud platforms, Kubernetes, and CI/CD systems, as well as experience with AI coding agents.
Job Description
About the Role
The DevOps Engineer - AI Model Evaluator role at mercor is a unique opportunity to work at the intersection of DevOps and AI. As a DevOps engineer in this role, you will be responsible for evaluating and improving the performance of AI models used in complex infrastructure engineering tasks. You will work closely with a team of elite creative and technical professionals to identify areas for improvement and implement solutions.The role is part of mercor's efforts to connect leading AI research labs with talented technical professionals. As a DevOps engineer in this role, you will have the opportunity to collaborate with a team of experts in the field and contribute to the development of cutting-edge AI technology.
Mercor is a company that values innovation and excellence, and this role is no exception. As a DevOps engineer in this role, you will be expected to bring a high level of expertise and passion to your work, and to be constantly looking for ways to improve and optimize the performance of AI models.
What You Will Do
- Evaluate the performance of AI models used in complex infrastructure engineering tasks
- Identify areas for improvement and implement solutions to optimize performance
- Collaborate with a team of elite creative and technical professionals to develop and implement AI models
- Use AI coding agents to complete and evaluate complex infrastructure engineering tasks
- Review model-generated implementations involving cloud platforms, Kubernetes, and CI/CD systems
- Identify bugs, edge cases, reliability issues, and failure modes in model outputs
- Compare outputs from multiple frontier models to assess strengths and weaknesses
- Apply professional engineering judgment to realistic infrastructure engineering scenarios
- Develop and maintain scripts and tools to automate tasks and improve efficiency
- Work closely with the development team to ensure smooth integration of AI models into the production environment
What We Are Looking For
- 2+ years of professional DevOps, SRE, or Cloud Engineering experience
- Experience with AWS, Azure, GCP, Kubernetes, Terraform, CI/CD pipelines, or observability tooling
- Regular use of AI coding agents like Cursor, Claude Code, Codex, Windsurf, Gemini CLI, or similar tools
- Ability to evaluate model-generated infrastructure and reliability engineering solutions
- Strong foundation in cloud platforms, Kubernetes, and CI/CD systems
- Experience with infrastructure automation and scripting languages like Python or Bash
- Strong problem-solving skills and attention to detail
- Excellent communication and collaboration skills
Nice to Have
- Experience supporting production-scale systems
- Knowledge of machine learning algorithms and deep learning architectures
- Experience with containerization using Docker
- Familiarity with agile development methodologies and version control systems like Git
Benefits and Perks
- Competitive compensation package
- Opportunity to work with cutting-edge AI technology
- Collaborative and dynamic work environment
- Flexible working hours and remote work options
- Access to a wide range of training and development resources
- Opportunities for career growth and professional development
- Recognition and rewards for outstanding performance
- Comprehensive health insurance and benefits package
- Generous paid time off and holidays
- Support for professional certifications and education
How to Stand Out
- Be prepared to provide examples of your experience with AI coding agents and cloud platforms.
- Make sure your resume and cover letter are tailored to the specific requirements of the role.
- Practice your problem-solving skills and be prepared to answer technical questions during the interview.
- Show enthusiasm and passion for working with AI technology and collaborating with a team of experts.
- Don't be afraid to ask questions during the interview to demonstrate your interest in the role and the company.
- Highlight your experience with infrastructure automation and scripting languages like Python or Bash.
- Prepare to discuss your experience with DevOps tools and methodologies, such as CI/CD pipelines and Kubernetes.
This is a remote position listed on WFA Digital, the platform for professionals who work from anywhere. Browse more remote jobs across all categories.