Staff AI Engineer - MCP Services
WFA Digital Insight
The demand for skilled AI engineers has grown significantly, with a 25% increase in job postings over the past year. As companies like Datadog invest heavily in AI-powered solutions, the need for experts who can develop and implement these technologies has never been higher. With its strong reputation for innovation, Datadog offers a unique opportunity for AI engineers to work on cutting-edge projects and collaborate with top talent. Before applying, candidates should be prepared to showcase their experience with AI systems, their ability to work in fast-paced environments, and their passion for shaping the future of AI in software development.
Job Description
About the Role
The Staff AI Engineer position at Datadog is a key role in the company's efforts to expand its MCP Services, a platform that enables external and internal agents to interact with Datadog data. As a Staff AI Engineer, you will be responsible for evolving the agent-friendly tool surface, driving the development of advanced evaluation frameworks for agentic systems, and contributing to the next generation of agent-tool interaction models. This is a fast-evolving space that requires someone who is eager to help shape Datadog's applied AI future.The MCP team is at the forefront of Datadog's innovation, and as a Staff AI Engineer, you will work closely with cross-functional teams to align on shared standards for tool integration and data access. Your expertise in applied AI, agentic programming, and AI system evaluation will be crucial in driving the success of this project.
Datadog's commitment to innovation and customer satisfaction has led to significant growth and recognition in the industry. As a Staff AI Engineer, you will be part of a talented team that is passionate about pushing the boundaries of what is possible with AI and software development.
What You Will Do
- Lead efforts to improve Datadog's public-facing MCP server, enabling intelligent agents to discover and interact with Datadog services
- Design and implement agentic tool surfaces tailored for evaluation and production use across a wide variety of AI agents
- Build and maintain advanced evaluation pipelines for measuring agent performance on Datadog workflows
- Investigate and resolve failure cases by analyzing tool output, improving query parsing, and enhancing agent feedback mechanisms
- Collaborate across Applied AI and internal teams to align on shared standards for tool integration and data access
- Develop and maintain technical documentation for AI tooling and systems
- Participate in code reviews and contribute to the improvement of the overall code quality
- Stay up-to-date with industry trends and advancements in AI and software development
- Contribute to the development of best practices for AI engineering within the company
- Work with the product team to develop and implement new features and functionalities
What We Are Looking For
- Experienced Staff-level engineer with a strong background in applied AI, agentic programming, and/or AI system evaluation
- Comfortable working in high ambiguity and fast-changing environments
- Skilled in building evaluation frameworks for LLM agents or AI systems, including metrics design and data instrumentation
- Strong systems thinking, able to reason across multiple agents, tools, and user scenarios
- Passionate about pushing boundaries in agent-augmented software and eager to shape evolving interfaces
- Demonstrated ability to use AI coding tools in day-to-day workflows and validate, critique, and refine AI-generated output
- Motivated to push the boundaries of how AI can improve software engineering best practices
- Experience working with the MCP standard or contributing to agent-compatible tooling surfaces
- Familiarity with building and evaluating ReAct agentic loops
Nice to Have
- Experience working with LLM-powered automation pipelines
- Familiarity with LLM orchestration frameworks
- Knowledge of agent orchestration and tool-use systems
- Experience with cloud-based AI platforms
- Certification in AI or software development
Benefits and Perks
- Generous and competitive benefits package
- New hire stock equity (RSUs) and employee stock purchase plan
- Continuous career development and pathing opportunities
- Employee-focused best-in-class onboarding
- Internal mentor and cross-departmental buddy program
- Friendly and inclusive workplace culture
- Flexible working hours and remote work options
- Access to cutting-edge technologies and tools
- Professional development opportunities and conference sponsorships
- Comprehensive health insurance and wellness programs
How to Stand Out
- Be prepared to showcase your experience with AI systems and your ability to work in fast-paced environments.
- Highlight your passion for shaping the future of AI in software development and your willingness to learn and adapt.
- Make sure to demonstrate your strong systems thinking and ability to reason across multiple agents, tools, and user scenarios.
- Emphasize your experience with evaluation frameworks for LLM agents or AI systems, including metrics design and data instrumentation.
- Be ready to discuss your experience with AI coding tools and your ability to validate, critique, and refine AI-generated output.
- Show your motivation to push the boundaries of how AI can improve software engineering best practices and contribute to building AI-enabled products.
- Research Datadog's products and services to understand how your skills and experience can contribute to the company's success.
This is a remote position listed on WFA Digital, the platform for professionals who work from anywhere. Browse more remote jobs across all categories.