Senior AI Engineer - APM Integrations

About the Role

The role requires close collaboration with other teams to understand their workflows and develop solutions that fit their processes. You will define the standards for these tools, measuring their effectiveness and continually improving them over time. Setting up evaluation and testing frameworks to ensure the accuracy and reliability of AI output will also be a key part of your responsibilities.

Datadog operates as a hybrid workplace, valuing both the flexibility of remote work and the creativity that comes from in-office collaboration. This role is remote-friendly, with opportunities available in Germany, Italy, and Spain, allowing you to work from the location that best suits your needs.

What You Will Do

• Develop AI-assisted tools that automate the integration process, from planning to implementation and validation
• Create systems that synthesize context from various sources to make informed changes that align with Datadog conventions and customer expectations
• Generate and evolve integration code and tests, including end-to-end scenarios that reflect real-world customer workloads and product features
• Design evaluation frameworks that prevent silent regressions, including golden sets, scenario baselines, semantic checks, performance thresholds, and release gating
• Build portfolio-level automation to proactively update for upstream breaking changes, rollout tracer features, migrate to new schemas or semantics, and expand coverage
• Partner with product managers, support engineers, and integration-owning teams to ensure the system is adoptable, trustworthy, and embedded in daily engineering workflows
• Define and track key metrics to measure the success and impact of AI-assisted tools
• Collaborate with cross-functional teams to align AI strategies with business objectives
• Stay up-to-date with the latest advancements in AI and machine learning, applying this knowledge to continuously improve AI-assisted tools
• Develop and maintain technical documentation for AI-assisted tools and processes

What We Are Looking For

• 6+ years of experience in building backend systems with a focus on simplicity, correctness, and performance
• Proven experience in delivering LLM/agent features to production, including prompting, tooling, evaluations, and safety/guardrails
• Strong understanding of machine learning fundamentals, including task definition, dataset construction, modeling, evaluation, deployment, and iteration
• Experience with distributed systems, including microservices performance, tracing, latency breakdowns, concurrency, and resiliency patterns
• Production operations mindset, with experience in monitoring, alerting, and participating in on-call rotations
• Ability to navigate ambiguity, iterate from prototype to production, and measure impact with clear metrics
• Experience with AI coding tools in day-to-day workflows and the ability to validate, critique, and refine AI-generated output
• Fluency with offline/online evaluations, including golden sets, automated regressions, and evaluation harnesses
• Solid grasp of statistics for experiments and the ML lifecycle

Nice to Have

• Hands-on experience with distributed tracing stacks, such as OpenTelemetry/Datadog APM, profilers, and logs/metrics pipelines
• Experience with planning/agent frameworks, tool-use orchestration, RAG, and retrieval/indexing over large context
• Experience building developer tools, such as IDEs, static analysis, compilers, and code transformation
• Knowledge of cloud computing platforms and containerization technologies

Benefits and Perks

• Competitive compensation package
• Opportunity to work remotely from Germany, Italy, or Spain
• Flexible working hours to accommodate different time zones and personal schedules
• Access to cutting-edge technologies and tools
• Collaborative and dynamic work environment
• Professional development opportunities, including training and conference attendance
• Comprehensive health insurance and wellness programs
• Generous parental leave policy
• Employee stock purchase plan

• When applying, make sure to highlight your experience with machine learning and AI-assisted tooling, as well as your ability to work in a distributed systems environment.
• Be prepared to discuss your approach to evaluating and refining AI-generated output, and how you ensure the accuracy and reliability of AI-assisted tools.
• Showcase your understanding of the ML lifecycle, from task definition to deployment and iteration, and your experience with statistics for experiments.
• Emphasize your ability to collaborate with cross-functional teams and communicate complex technical concepts to non-technical stakeholders.
• Consider creating a portfolio that demonstrates your experience with AI-assisted tooling and integration development, including any personal projects or contributions to open-source repositories.
• During the interview process, ask about the company's approach to AI strategy and how the role contributes to the company's overall goals and objectives.