Senior Software Engineer, AI

About the Role

The role of a Senior Software Engineer, AI is high-ownership, with a significant impact on shaping evaluation methodology, agent architecture, and the core systems that determine AI performance in production. This includes working on traces, evaluations, and human-in-the-loop feedback loops connected across all of Lattice’s AI use cases.

What You Will Do

• Design and ship a robust, end-to-end AI evaluation framework covering offline evaluations, production tracing, and human-in-the-loop feedback loops.
• Define and instrument key metrics such as agent task completion rates, hallucination rates, response quality, user engagement, and downstream business outcomes.
• Build and maintain evaluation datasets, test harnesses, and automated scoring pipelines to catch regressions before they ship.
• Identify and surface the drivers of agent quality improvement, giving the team clear signals on where to invest.
• Architect and implement reusable agent infrastructure, including multi-turn conversation workflows, recommendation services, LLM DAGs, and standardized agent topology patterns using LangGraph.
• Build and scale RAG pipelines and retrieval infrastructure, including vector store management and retrieval quality optimization.
• Make principled build vs. buy decisions across LLM providers, agent frameworks, and evaluation tooling, balancing capability, cost, latency, and vendor risk.
• Contribute to production AI systems with a strong focus on reliability, observability, and performance, not just prototypes.
• Own projects end-to-end, driving them to completion and bringing in the right people at the right time.
• Partner with engineering leads and managers to inform technical direction on agent quality and evaluation strategy.
• Raise the AI engineering bar across the broader team through code review, documentation, and thoughtful technical debate.

What We Are Looking For

• 5+ years of professional software engineering experience with significant time spent on production AI/ML systems.
• Deep hands-on experience with LLM-based systems, including prompt engineering, RAG pipelines, agent orchestration, evaluation metrics, and model fine-tuning.
• Proven ability to work with data and understand statistics, especially in experiments.
• Proven ability to build and operate agentic AI systems in production, including multi-step workflows and multi-agent topologies.
• Strong command of AI evaluation, having built eval frameworks before and understanding the difference between a good eval and a vanity metric.
• Production-grade Python engineering, with an emphasis on clean, maintainable, testable code.
• Experience with LangGraph or comparable agent orchestration frameworks.
• Familiarity with LangSmith or comparable LLM observability tooling for tracing, evaluation, and debugging.

Nice to Have

• Experience with cloud platforms such as AWS or GCP, particularly in deploying and managing AI/ML workloads.
• Knowledge of containerization using Docker and orchestration using Kubernetes.
• Experience with agile development methodologies and version control systems like Git.
• Participation in open-source projects or personal projects that demonstrate AI/ML engineering skills.

Benefits and Perks

• Competitive salary package.
• Opportunity to work on cutting-edge AI technologies and contribute to the development of innovative products.
• Collaborative and dynamic work environment with a team of experienced engineers and researchers.
• Flexible working hours and remote work options, allowing for a great work-life balance.
• Access to professional development opportunities, including training, conferences, and workshops.
• Comprehensive health insurance and retirement plans.
• Generous parental leave policy and family support benefits.
• Onboarding program and mentorship to help you get started and succeed in your role.

• To stand out, emphasize your experience with production AI/ML systems and LLM-based technologies in your resume and cover letter.
• Prepare to discuss your approach to AI evaluation and how you have implemented it in previous roles.
• Showcase your ability to work independently and lead projects, highlighting any successes where you drove a project from conception to completion.
• Be ready to explain technical concepts and decisions to both technical and non-technical stakeholders, demonstrating your communication skills.
• Consider including a portfolio or GitHub link with examples of your AI/ML projects, especially those related to agent architecture and evaluation frameworks.
• Research the company culture and values to understand how your skills and experience align with Lattice's mission and goals.