Test Analyst, AI

The AI Test Analyst is responsible for ensuring the quality, reliability, and effectiveness of AI-powered solutions, including generative AI applications, intelligent agents, and AI-enabled business processes. The role focuses on validating AI models and outputs, assessing data quality, monitoring performance, and ensuring adherence to Responsible AI principles, regulatory requirements, and organizational standards.

The role supports the adoption of AI-driven testing practices by leveraging automation, innovative testing techniques, and AI-enabled tools to improve testing efficiency and coverage. It also contributes to the development of testing capabilities across the organization by promoting best practices and supporting the upskilling of testing teams in AI and automation technologies.

As MUFG Retirement Solutions continues to integrate AI across its products, services, and testing functions, the AI Test Analyst plays a key role in embedding AI into testing processes, identifying cost-effective quality assurance solutions, and ensuring that testing strategies align with business objectives, quality standards, and the organization's AI transformation agenda.

Key Accountabilities and main responsibilities

Strategic Focus

• Develop and execute testing strategies for AI, Machine Learning (ML), and Generative AI solutions to ensure accuracy, reliability, performance, and business value.
• Establish and maintain test automation frameworks and scripts that support various types of testing, such as functional testing, integration testing, API testing, system testing, model validation, data validation, API testing, system testing, and end-to-end AI solution testing.
• Validate AI model outputs against defined acceptance criteria, business requirements, and responsible AI standards.
• Ensure comprehensive testing coverage across the AI lifecycle, including data ingestion, preprocessing, model training, inference, output validation, and reporting.
• Assess AI solution performance across dimensions such as accuracy, relevance, consistency, explainability, fairness, robustness, and reliability.
• Integrate AI testing activities and test automation framework & scripts with continuous integration, CI/CD pipelines and tools to support continuous testing and deployment of AI solutions.
• Collaborate with developers, business analysts, product owners, and project managers to define testing requirements, success criteria, and quality measures.
• Manage the execution of automated test suites, monitor the test results and collaborate with teams to troubleshoot and resolve the issues promptly.
• Identify opportunities to leverage AI-driven testing tools and automation techniques to improve testing efficiency and effectiveness.
• Research emerging AI testing methodologies, tools, technologies, new test automation tools and industry best practices, providing recommendations for continuous improvement.
• Support the establishment and adoption of enterprise standards, processes, and quality controls for AI testing and validation.

Operational Management

• Design, develop, execute, and maintain test cases and test scripts for AI, ML, and Generative AI applications.
• Validate data quality, data integrity, and data lineage across AI systems and supporting platforms.
• Perform functional, integration, API, regression, user acceptance, and end-to-end testing of AI-enabled solutions.
• Evaluate AI model outputs for accuracy, consistency, bias, hallucinations, and compliance with business requirements and existing processes.
• Execute prompt testing, response validation, and scenario-based testing for Generative AI solutions.
• Monitor test execution results, analyse outcomes, and produce accurate and comprehensive test reports.
• Identify, document, track, and support resolution of defects, model performance issues, and data quality concerns.
• Collaborate with business analysts, developers, and other stakeholders to understand requirements and deliver high-quality AI solutions.
• Support test data preparation, environment validation, and model release verification activities.
• Contribute to continuous improvement initiatives by identifying opportunities to enhance AI testing frameworks, processes, and automation capabilities.
• Provide knowledge sharing and support to project teams regarding AI testing approaches, tools, and best practices.

Governance & Risk

• Identify quality, model, data, security, privacy, ethical, and compliance risks associated with AI solutions and proactively support mitigation activities.
• Validate adherence to Responsible AI principles, regulatory requirements, organizational policies, and governance standards.
• Support the assessment and reporting of AI-related risks, including model bias, explainability, fairness, and data quality concerns.
• Maintain appropriate test evidence, validation reports, defect records, and audit documentation to support governance and compliance requirements.
• Escalate risks, issues, and control gaps that may impact AI solution quality, regulatory compliance, or business outcomes.

The above list of key accountabilities is not an exhaustive list and may change from time-to-time based on business needs.

Experience & Personal Attributes

Experience

• 3+ years of experience in Quality Engineering, Test Automation, and AI-driven testing, with preference for candidates having 2+ years of experience in Retirement Solutions, Financial Services, or BFSI domains.
• Strong experience in designing, developing, and executing test automation solutions using tools such as Selenium, Appium, UFT/OpenText (at least one mandatory), Python, Maven, Postman, REST APIs, SoapUI, Cortex, and Snowflake.
• Hands-on experience with AI-assisted testing tools and technologies for test case generation, test optimization, defect prediction, intelligent test execution, quality analytics, and test automation.
• Experience in validating AI/ML-enabled applications, including model testing, data quality validation, prompt testing, response evaluation, hallucination detection, bias and fairness testing, explainability validation, adversarial testing, and assessment of AI model accuracy, reliability, and performance.
• Strong understanding of software testing principles, methodologies, and best practices, including test planning, test design, test execution, defect management, reporting, governance, and quality assurance processes.
• Solid understanding of SDLC and SQA processes across Agile, Waterfall, and DevOps delivery models.
• Experience working with CI/CD pipelines and DevOps practices using Jenkins, Azure DevOps, Azure Repos, Bitbucket, Git, Docker, and Kubernetes.
• Strong analytical and problem-solving skills with the ability to define quality standards, automate validation processes, and ensure reliable, scalable, and secure AI-driven applications.
• Experience with cloud platforms, data validation, API testing, integration testing, and end-to-end testing across complex enterprise environments.
• Design, develop, and execute comprehensive test strategies, test plans, and test cases for functional, integration, system, regression, API, and end-to-end testing across applications, data, and reporting layers.
• Collaborate closely with Business Analysts, Developers, Product Owners, and other stakeholders to define test scenarios, automate test cases, and ensure quality throughout the software development lifecycle.
• Develop and maintain automated testing solutions leveraging both traditional and AI-powered testing tools to improve testing efficiency, coverage, and quality.
• Validate AI-powered applications, intelligent agents, and machine learning models by assessing model accuracy, reliability, performance, fairness, explainability, and compliance with Responsible AI principles.
• Perform prompt testing, response validation, hallucination detection, adversarial testing, and evaluation of AI outputs against business requirements and expected outcomes.
• Execute API, data, integration, and end-to-end testing to ensure data integrity, system interoperability, and reliable business process outcomes.
• Manage defects and quality issues using tools such as JIRA and Azure DevOps, including defect tracking, root-cause analysis, reporting, and resolution support.
• Produce test evidence, quality metrics, and test execution reports to support audit, compliance, and governance requirements.
• Contribute to the continuous improvement of testing frameworks, methodologies, and automation practices, identifying cost-effective solutions and opportunities for innovation.
• Provide guidance, support, and training to team members on AI-enabled testing tools, automation technologies, and quality engineering best practices.

Personal Attributes

• Strong analytical and problem-solving skills
• Attention to detail and quality-driven mindset
• Good communication and teamwork skills
• Proactive and eager to learn