Consultant - Test Engineer_Nexus AI

Test Engineer, AI Product Quality (US Healthcare Payer) Role Summary: We are seeking a highly motivated and skilled Test Engineer with 3-6 years of experience to join our team, focusing on ensuring the quality, reliability, and performance of our innovative native AI-driven products within the US healthcare payer platform. This role requires a strong foundation in software testing principles, a keen eye for detail, and a proactive mindset in approaching the unique challenges of AI/ML model validation and integration testing. You will collaborate closely with product managers, data scientists, and developers to deliver high-quality, impactful AI solutions to our members and internal stakeholders. • AI Product Quality Assurance: Specializes in the comprehensive testing and validation of AI/ML-powered applications and components, ensuring they meet functional, non-functional, and ethical requirements within the sensitive US healthcare payer context. • Healthcare Payer Domain Expertise: Leverages a strong understanding of healthcare payer operations to design relevant test scenarios and validate AI solutions' impact on claims, member services, provider networks, and regulatory compliance. • Proactive Test Strategy & Design: Contributes to the development of robust test strategies, designs detailed test plans and cases, and proactively identifies potential failure points and data quality issues critical for AI/ML performance. • Automation & Efficiency: Drives the adoption and implementation of test automation frameworks and scripts, particularly for data validation, API testing, and AI model inference testing, to enhance efficiency and coverage. Builds reusable AI evaluation harnesses, synthetic test datasets, regression suites for prompts/agents, CI/CD integrated AI evaluation checks. • Collaborative & Communicative: Acts as a vital link between technical development teams (Data Scientists, Developers) and business stakeholders, clearly communicating test results, defects, and risks related to AI product quality. Partners with SMEs, Product, Data Science, Engineering, Architecture, AI Governance and Operations to review AI outputs, knowledge gaps, model drift and release readiness Essential Responsibilities: • Test Strategy & Planning: Work closely with Product Managers, Data Scientists, and Developers to understand AI product requirements, model functionalities, and system architecture. Develop comprehensive test strategies and detailed test plans for AI-driven features and end-to-end solutions. • Test Case Design & Execution: Design, develop, and execute a wide range of test cases, including functional, integration, regression, performance, and user acceptance tests, specifically for AI/ML models and their integration into existing payer systems. • AI/ML Model Validation Support: • Contribute to the validation of AI/ML model outputs, focusing on data quality, data drift, bias detection, and performance against defined metrics. • Design tests to evaluate model behavior under various data conditions and edge cases. • Work with Data Scientists to understand model limitations and interpretability, translating these into test scenarios. • AI Solution Quality & Governance Testing: Design and execute end-to-end testing of AI-driven applications, workflows, and data processes to ensure accuracy, reliability, usability, and compliance. Validate information quality, system behavior, approval workflows, auditability, and operational performance while ensuring seamless integration across business and technology components. • Test Automation Development: Design, develop, and maintain automated test scripts and frameworks for API testing, UI testing, data validation, and automated model inference testing using relevant tools and programming languages (e.g., Python, Java, Selenium, Postman). • Defect Management & Reporting: Identify, document, and track defects with clear, concise, and reproducible steps. Collaborate with development teams to ensure timely resolution and retesting. Provide regular, clear reports on test progress, defect status, and overall product quality to project stakeholders. • Data Validation & Quality: Focus on testing data pipelines and data transformations that feed AI models, ensuring data accuracy, completeness, and consistency. (Optional but highly valued: Design and execute ETL tests). • Performance & Scalability Testing: Participate in performance and scalability testing for AI applications, ensuring they meet defined service level agreements (SLAs) under expected load. • Continuous Improvement: Proactively identify areas for test process improvement, recommend tools and techniques, and contribute to the enhancement of testing methodologies, particularly for AI product quality assurance. • Documentation: Create and maintain clear, comprehensive test documentation, including test plans, test cases, and test reports. Required Skills & Experience: • Professional Experience (3-6 years): • 3-6 years of progressive experience in software testing or quality assurance roles. • Minimum 2-3 years directly testing applications or products within the US healthcare payer industry, demonstrating a strong understanding of claims processing, member management, provider data, and regulatory compliance (e.g., HIPAA). • AI/ML Testing Acumen: • Foundational understanding of Artificial Intelligence (AI) and Machine Learning (ML) concepts, including model types, data requirements, and the unique challenges of testing AI (e.g., non-determinism, data dependency, bias). • Experience in testing AI/ML-powered features or products is highly preferred, even if informal. • Ability to think proactively about how to test the 'intelligence' and 'behavior' of AI models, not just their functional interfaces. • Test Design & Methodologies: • Proven expertise in designing comprehensive test plans and detailed test cases based on business requirements and technical specifications. • Strong understanding of various testing methodologies (Agile, Waterfall) and different types of testing (functional, non-functional, integration, regression, UAT). • Strong understanding of AI testing layers: data/knowledge layer testing, model testing, agent testing, application/workflow testing, end-to-end AI system testing, and responsible AI/governance testing. • Automation Experience: • Hands-on experience in developing and maintaining automated test scripts using programming languages like Python or Java. • Familiarity with test automation frameworks and tools (e.g., Selenium, Playwright, Cypress, Postman for API testing). • Technical Skills: • Solid proficiency in SQL for data validation, querying databases, and analyzing test data. (Minimum 2 years' experience). • Experience with API testing tools (e.g., Postman, SoapUI). • Familiarity with version control systems (e.g., Git). • Foundational understanding of GCP/Vertex AI, embeddings/vector stores, knowledge graphs/Neo4j or equivalent graph solutions, • Communication & Collaboration: • Excellent verbal and written communication skills, with the ability to clearly articulate technical issues, test results, and risks to diverse audiences, including technical and non-technical stakeholders. • Strong interpersonal skills and proven ability to collaborate effectively within cross-functional teams (Product, Data Science, Engineering). • Problem-Solving & Analytical Skills: • Exceptional analytical skills to diagnose complex issues, identify root causes, and propose effective testing solutions. • A meticulous and detail-oriented approach to quality assurance. Desired Qualifications (Optional but a strong plus): • Experience with ETL testing and data warehousing concepts. • Familiarity with cloud platforms (e.g., AWS, GCP, Azure) and testing applications deployed in the cloud. • Knowledge of performance testing tools (e.g., JMeter, LoadRunner). • Experience with CI/CD pipelines and integrating automated tests. ..............