Senior Software Engineer II Applied AI and Evaluations

SmartsheetSmartsheet·Remote·Work From Anywhere
Software Development

WFA Digital Insight

As the demand for AI-powered solutions grows, companies like Smartsheet are investing heavily in developing cutting-edge technologies. With a 25% increase in AI adoption over the past year, professionals with expertise in AI evaluation and prompt engineering are in high demand. Smartsheet's commitment to innovation and employee growth makes this role an attractive opportunity for those looking to make a meaningful impact. Before applying, candidates should be aware that this position requires a deep understanding of LLM evaluation and retrieval-augmented generation, as well as experience working with Databricks and MLflow.

Job Description

About the Role

The Senior Software Engineer II role at Smartsheet is a unique opportunity to work at the intersection of AI, software engineering, and product development. As a key member of the SmartAssist team, you will be responsible for driving the quality of AI-powered agents, ensuring they meet the highest standards of performance and reliability. Your work will have a direct impact on the success of Smartsheet's customers, enabling them to automate manual tasks, uncover insights, and scale their operations more efficiently.

The SmartAssist platform is designed to empower teams to work more effectively, and as a Senior Software Engineer II, you will play a critical role in shaping the future of this technology. You will collaborate closely with cross-functional teams, including Agent Engineering and AI Platform, to design and implement solutions that address complex technical challenges. Your expertise in AI evaluation, prompt engineering, and regression testing will be essential in driving the development of high-quality agents that meet the evolving needs of Smartsheet's customers.

Smartsheet is committed to fostering a culture of innovation, experimentation, and continuous learning. As a Senior Software Engineer II, you will have the opportunity to work with cutting-edge technologies, participate in hackathons, and collaborate with talented professionals who share your passion for AI and software engineering.

What You Will Do

  • Own the end-to-end quality of AI-powered agents, including diagnosis, improvement, and validation
  • Identify failure modes across quality dimensions, such as factual accuracy, completeness, tone, actionability, and latency
  • Design and implement prompt engineering solutions to drive quality improvements
  • Collaborate with Agent Engineering and AI Platform teams to develop and refine the SmartAssist platform
  • Work closely with the AI Platform team to integrate with Databricks and MLflow
  • Develop and maintain tools and systems to support agent development, testing, and deployment
  • Participate in code reviews, ensuring that all code meets the highest standards of quality and maintainability
  • Stay up-to-date with industry trends, best practices, and emerging technologies in AI and software engineering
  • Contribute to the development of the Agent Development Lifecycle (ADLC) and ensure that all agents meet the required quality standards

What We Are Looking For

  • 5+ years of experience in software engineering, with a focus on AI, machine learning, or natural language processing
  • Strong proficiency in programming languages such as Python, Java, or C++
  • Experience with AI evaluation, prompt engineering, and regression testing
  • Knowledge of Databricks, MLflow, or similar technologies
  • Excellent problem-solving skills, with the ability to analyze complex technical issues and develop creative solutions
  • Strong collaboration and communication skills, with experience working with cross-functional teams
  • Bachelor's or Master's degree in Computer Science, AI, or a related field
  • Experience working in an Agile development environment, with familiarity with Scrum or Kanban methodologies

Nice to Have

  • Experience with cloud-based technologies, such as AWS or Google Cloud
  • Knowledge of containerization using Docker or Kubernetes
  • Familiarity with DevOps practices, including continuous integration and continuous deployment
  • Experience with data analytics, data visualization, or data science

Benefits and Perks

  • Competitive salary and benefits package
  • Opportunity to work with cutting-edge technologies and innovative products
  • Collaborative, dynamic work environment with a talented team of professionals
  • Flexible working hours, remote work options, and a healthy work-life balance
  • Access to professional development opportunities, including training, mentorship, and conference attendance
  • Recognition and rewards for outstanding performance and contributions to the company's success
  • Comprehensive health insurance, retirement savings plan, and other benefits
  • Generous paid time off, including vacation, sick leave, and holidays
  • Employee referral program with rewards for successful referrals

How to Stand Out

  • Be prepared to provide specific examples of your experience with AI evaluation, prompt engineering, and regression testing.
  • Showcase your problem-solving skills by walking the interviewer through your thought process and approach to complex technical challenges.
  • Highlight your collaboration and communication skills, emphasizing your ability to work effectively with cross-functional teams.
  • Demonstrate your knowledge of industry trends, best practices, and emerging technologies in AI and software engineering.
  • Be prepared to discuss your experience with Databricks, MLflow, or similar technologies, and how you have applied them in previous roles.
  • Show enthusiasm for Smartsheet's mission and products, and be prepared to discuss how your skills and experience align with the company's goals and values.

This is a remote position listed on WFA Digital, the platform for professionals who work from anywhere. Browse more remote jobs across all categories.