Senior QA Automation Engineer (AI & LLM Testing)

About the Role

As part of Nagarro's team, the Senior QA Automation Engineer will be working in an agile environment, contributing to refinement sessions, sprint planning, retrospectives, and daily collaboration. The role involves not only technical expertise but also the ability to communicate complex test strategies and quality risks to both technical and non-technical stakeholders. Nagarro values a non-hierarchical culture, making this role an excellent opportunity for someone who enjoys collaborative work and innovation.

Nagarro is expanding its digital and AI solutions for businesses across various sectors, including financial services, healthcare, consumer goods, technology, energy, and travel. This growth positions the company as a leader in digital product engineering, with a need for skilled professionals who can drive quality and innovation in AI and LLM testing.

What You Will Do

• Design, develop, and maintain automated test suites for web applications, APIs, and AI-powered features.
• Evaluate LLM outputs for response accuracy, relevance, consistency, and hallucination risks.
• Create prompt tests and quality checks using Promptfoo or similar LLM evaluation frameworks.
• Build end-to-end UI automation using Playwright, following page object models and maintainable test architecture principles.
• Develop reusable Python scripts for API testing, automation utilities, and LLM quality evaluation.
• Create and maintain Postman collections for REST API testing, including environments, assertions, and CI/CD integration.
• Utilize GitHub Copilot or similar AI coding assistants to improve test coverage, speed up automation development, and identify quality gaps.
• Collaborate closely with product managers, developers, and stakeholders to define test strategies, communicate risks, and improve quality practices.
• Actively contribute to Agile ceremonies, including refinement sessions, sprint planning, retrospectives, and daily collaboration.
• Develop and maintain technical documentation of test suites and test environments.
• Participate in the development of the company's testing standards and best practices.

What We Are Looking For

• Strong experience in QA Automation and software testing (5+ years).
• Hands-on experience testing AI or LLM-based features, including prompt testing, output validation, response accuracy, and hallucination mitigation.
• Experience with Promptfoo or similar LLM evaluation tools/frameworks.
• Strong proficiency in Playwright for end-to-end UI automation.
• Solid knowledge of page object patterns, reusable test components, and scalable test architecture.
• Strong Python scripting skills for automation, API helpers, and custom quality checks.
• Experience with Postman and REST API testing, including collections, assertions, environments, and CI/CD integration.
• Practical experience using GitHub Copilot or similar AI coding assistants.
• Experience working in Agile environments such as Scrum or Kanban.
• Strong communication skills, with the ability to explain test strategies and quality risks to technical and non-technical stakeholders.
• Ability to work in a remote team environment.
• Strong analytical and problem-solving skills.

Nice to Have

• Experience with mobile test automation using Appium, XCUITest, or similar tools.
• Understanding of iOS and Android platform-specific testing considerations.
• Experience integrating automated tests into CI/CD pipelines.
• Familiarity with MCP or agent-assisted testing approaches in Playwright.
• Knowledge of Docker and containerization for test environments.

Benefits and Perks

• Competitive compensation package.
• Opportunity to work with a global digital product engineering company.
• Collaborative, non-hierarchical work environment.
• Professional development opportunities through training and workshops.
• Access to the latest technologies and tools in AI and LLM testing.
• Flexible working hours and remote work options.
• Health insurance and other benefits.
• Annual leave and paid holidays.
• Continuous learning and education support.
• Recognition and reward for outstanding performance.

• Ensure you have a solid portfolio showcasing your experience in QA automation, especially in AI and LLM testing.
• Practice explaining complex technical concepts to non-technical individuals, as this is a key aspect of the role.
• Familiarize yourself with the latest tools and frameworks in AI and LLM evaluation, such as Promptfoo.
• Highlight any experience working in agile environments and contributing to agile ceremonies.
• Prepare to discuss your approach to test automation, including how you maintain and scale test suites.
• Show enthusiasm for continuous learning, as the field of AI and LLM testing is rapidly evolving.
• Prepare questions to ask the interviewer about the company culture, team dynamics, and opportunities for growth.