Associate Consultant

AI Test Engineer

Responsibilities / Job Description

• Design and execute test strategies for machine learning models (classification, regression, NLP, CV, etc.). • Validate model accuracy, fairness, bias detection, explainability, and robustness under varying datasets. • Perform adversarial testing and edge-case validation for AI systems. • Test autonomous agents, multi-step reasoning paths, and state transitions. • Validate correctness, grounding, consistency, and safety of LLM outputs. • Evaluate prompt robustness and behavioral variations across scenarios. • Validate retrieval accuracy, grounding quality, and hallucination reduction. • Test vector store behavior, document chunking logic, and retriever configurations. • Validate fallback behaviors when tools or external services fail. • Execute adversarial, prompt-injection, and red team style testing. • Validate compliance with data privacy, business rules, and insurance guidelines. • Benchmark LLM latency, throughput, and multi-agent performance. • Validate concurrency handling, degradation logic, and retry mechanisms. • Document test cases, behavior maps, prompt variations, and evaluation reports. • Work closely with AI engineers and SMEs to refine agent workflows. • Functional & Non-Functional Testing: o Conduct API testing for AI services and model endpoints. o Validate performance, scalability, and latency of AI inference services. o Perform security testing for data pipelines and model-serving endpoints. • Quality Governance & Compliance: o Ensure compliance with AI ethics, data privacy, and regulatory standards. o Establish KPIs for model quality (precision, recall, F1-score, drift detection). o Document test strategies, results, and provide audit-ready evidence. • Collaboration & Leadership: o Work closely with data scientists, ML engineers, and DevOps teams to integrate testing into the AI lifecycle. o Mentor junior testers and contribute to building reusable test accelerators. o Represent the testing function in client discussions, providing insights on AI quality assurance. • Develop automated test frameworks for AI/ML pipelines. • Implement synthetic data generation for edge-case testing. • Conduct bias and fairness testing across diverse datasets. • Validate monitoring dashboards for model drift and performance degradation. • Perform regression testing for retrained models. • Create reusable test assets for AI projects (scripts, datasets, frameworks). • Provide detailed defect analysis and SLA-driven closure.

Required Skills

• Strong experience in software testing (manual + automation) with focus on AI systems. • Hands-on expertise in Python for test automation, data validation, and AI testing scripts. • Knowledge of Gen AI frameworks (Langchain, Langgraph, CrewAI). • Familiarity with CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI). • Strong understanding of data validation, model evaluation metrics, and drift detection. • API testing experience (REST, GraphQL) with tools like Postman or REST Assured. • Solid knowledge of cloud platforms (AWS, Azure, GCP) for AI deployments. • Exposure to AI ethics and compliance testing. • Knowledge of containerization & orchestration (Docker, Kubernetes). • Experience with Big Data ecosystems (Spark, Hadoop) for large-scale AI testing. • Familiarity with observability tools (Prometheus, Grafana, ELK stack). • Understanding of security testing for AI pipelines (data poisoning, adversarial attacks).