About the Role
We are looking for an AI Tester to ensure the quality, reliability, and safety of Generative AI and Large Language Model (LLM)-based solutions. The role will focus on functional, prompt, response-quality, automation, API, regression, performance, and safety testing of enterprise AI/GenAI applications.
The candidate will work closely with AI Engineers, Data Scientists, MLOps teams, Product Architects, and business stakeholders to validate AI solutions and ensure they are accurate, reliable, safe, and production-ready.
Key Responsibilities / Requirements
Define AI/GenAI test strategies and scope for LLM applications, including Azure OpenAI and Azure AI Foundry.
Design and execute functional, prompt, and response-quality test cases.
Validate accuracy, relevance, hallucinations, bias, fairness, toxicity, and AI safety.
Perform regression testing across AI/ML models, applications, and model versions.
Build and maintain automated test suites using Python and PyTest, with Selenium/Playwright where applicable.
Perform API and integration testing using Postman and REST APIs.
Conduct performance and latency testing of AI/LLM services.
Log, track, and manage defects using Jira.
Prepare quality reports, test documentation, and actionable insights.
Improve test coverage, automation, tooling, and testing processes.
Work effectively in Agile environments across multiple teams and technologies.
Collaborate with AI Engineers, Data Scientists, Product Architects, and business stakeholders.
Contribute to test strategies, quality roadmaps, standards, and governance practices.
Strong Python programming skills with hands-on PyTest experience.
Experience with REST APIs, Postman, Selenium/Playwright, SQL, Git, and CI/CD pipelines.
Strong understanding of AI/ML regression testing, hallucination detection, bias/fairness, toxicity/safety, and response-quality evaluation.
Bachelor's degree in Computer Science, Engineering, or a related technical field.
Strong communication, stakeholder management, problem-solving, and independent working skills.
Good understanding of Agile and Waterfall methodologies.
Good to Have
Hands-on experience with Azure OpenAI / Azure AI Foundry.
Familiarity with LLM evaluation frameworks such as RAGAS, DeepEval, TruLens, or Azure AI Evaluation SDK.
Experience with prompt engineering for test design.
Exposure to MLOps tools such as MLflow or Kubeflow.
Experience working with MLOps and AI Engineering teams.