AI QA Test Lead

Truliant's mission is to improve lives by putting our members first, providing great service and straightforward financial solutions. Our core values of Member Focus, Service, Guidance, Relationships, and Community define how we interact with members to fulfill our mission of improving lives and providing a foundation on which we conduct ourselves.

Our Promise to Our Teammates: Truliant strives to provide all teammates with an enjoyable place to work where they feel valued, empowered and rewarded for all that they do.

Purpose of the Job

The AI QA Test Lead is a critical quality and risk-management leader responsible for ensuring the reliability, accuracy, and production readiness of the credit union's AI-driven member and staff experiences. This role owns the overall AI quality assurance strategy, directly safeguarding millions of AI-powered interactions each year by ensuring solutions meet defined standards for accuracy, compliance, brand alignment, and ethical use. Serving as the primary authority on AI testing and validation, the AI QA Test Lead designs and manages comprehensive testing plans across conversational AI, generative AI, and agentic AI workflows. The role partners closely with the AI Program team and Truliant's business stakeholders, and key vendors to validate new use cases, manage AI-related risk, and ensure all AI releases are thoroughly tested before deployment. The role manages the end-to-end correction loop with third-party vendors integral to the delivery of Truliant's AI use cases. Key responsibilities include defining QA plans, leading end-to-end UAT, validating ethical guardrails, and establishing scalable QA practices that enable safe and confident expansion of Truliant's AI capabilities.

Essential Functions and Responsibilities

• Owns and leads the end-to-end quality assurance strategy for the credit union's AI self-service and internal support solutions, with a primary focus on conversational AI platforms such as Kore.ai.

• Defines, documents, and maintains AI QA standards, test frameworks, methodologies, and acceptance criteria across all AI initiatives.

• Designs and executes comprehensive AI test plans covering functional accuracy, conversational flows, edge cases, integrations, compliance, and regression scenarios.

• Leads and governs all AI-related user acceptance testing (UAT), serving as the final quality gate for approving AI solutions prior to production release.

• Validates AI-generated responses for accuracy, consistency, brand alignment, and adherence to member-first service standards.

• Tests and monitors ethical AI guardrails, including bias prevention, hallucination detection, inappropriate response handling, and escalation logic.

• Partners closely with the AI Solutions Architect and AI Optimization Analyst to review solution designs and ensure testability throughout the AI development lifecycle.

• Establishes testing processes for prompt changes, workflow updates, model enhancements, and new AI capabilities to ensure safe iteration and deployment.

• Analyzes defects, testing outcomes, and production issues to identify root causes and drive continuous improvement in AI quality.

• Collaborates with business units and frontline teams to validate AI behavior against real-world use cases and operational expectations.

• Tracks, documents, and reports on AI quality metrics such as defect trends, release readiness, risk severity, and post-release issues.

• Stays current on AI QA best practices, regulatory considerations, and emerging testing techniques, proactively recommending improvements to strengthen AI reliability and governance.

• Should possess a strong commitment to providing excellent service to Truliant's members.

Other Duties and Responsibilities

• Assists with other tasks and projects as assigned.

Knowledge, Skills, and Abilities

• Must have a strong understanding of conversational AI, generative AI, NLP, and intent-based bot frameworks.

• Must have the ability to evaluate AI responses for tone, brand voice, compliance, and member experience impact.

• Must have the ability to understand credit union business processes and assess AI solutions against operational and member needs.

• Must have knowledge of ethical AI principles, including bias detection, guardrail validation, and hallucination mitigation.

• Must have excellent collaboration and communication skills to work across technical teams, business stakeholders, and frontline staff.

• Must have the ability to translate business and regulatory requirements into clear AI testing criteria and validation scenarios.

• Must have familiarity with analytics, dashboards, automated QA tools, or conversation intelligence platforms used to support testing and monitoring.

• Must have strong problem-solving skills with the ability to identify systemic quality issues and propose scalable solutions.

• Must have attention to detail and high standards when reviewing AI outputs, workflows, and system changes.

• Must have an understanding of UX and conversational design principles as they relate to testing and validation.

• Must have the ability to manage multiple priorities in a fast-paced, innovation-driven environment.

• Must have strong organizational skills and the ability to work independently with minimal supervision.

• Must be able to work in a team environment with strong collaborative skills.

• Must have superior interpersonal skills; ability to get along with diverse personalities; tactful, mature, flexible, respectful.

• Must be flexible and able to shift resources and priorities as required.

• Should possess a strong commitment to providing excellent service to Truliant's members.

Physical Requirements

• Occasional standing, walking, bending, and stooping required

• Must be able to sit at a desk for long periods of time and use a computer

• Must be able to moderately lift or move up to 5 pounds and occasionally lift or move up to 10 pounds

Education and Background

• Bachelor's degree required in a relevant field such as Information Systems, Computer Science, Business Administration, or a related discipline required.

• Equivalent combination of education and directly applicable work experience may be considered, particularly in AI QA, software testing, or risk-focused roles.

• 3-7 years of experience in quality assurance, testing, or validation roles involving complex systems, with at least 2 years focused on AI, conversational AI, or automation platforms required.

• Demonstrated experience leading test planning, execution, defect management, and UAT for enterprise solutions required.

• Expertise in quality assurance methodologies, test planning, test execution, defect tracking, release validation, and vendor correction loops required.

• Experience testing AI systems for accuracy, edge cases, failure modes, and unintended behavior required.

• Strong experience leading UAT efforts and serving as the final approval authority for production releases required.

• Experience testing AI or automated decision systems, including generative AI or NLP-based solutions, strongly preferred.

• Experience partnering with AI engineers, solutions architects, product owners, or business leaders to ensure quality and risk alignment preferred.

• Background in financial services, credit unions, banking, or other regulated environments is a plus.

This position is remote within our NC, SC, VA footprint. Although this position is primarily remote, there are occasional events and meetings that may require you to workon-site at one of our Truliant locations.

If you have a passion for member service and would like to invest in a meaningful career with opportunity for growth, we encourage you to apply to Truliant!

As a member of the Truliant family, you will enjoy the following full time benefits:

• No-cost employee Medical, dental, vision coverage
• Prescription benefits (including mail order)
• Paid holidays and Paid Time Off (PTO)
• 401(k) plan with contribution matching
• Paid community involvement volunteers hours
• Paid group life Insurance
• Teammate loan discounts
• Tuition reimbursement
• Short and long-term disability
• Health & Wellness program
• Teledoc (physician video conferencing)
• Onsite fitness facilities or health club reimbursement
• Employee Assistance Program (EAP)
• Medical flexible spending account
• Dependent care flexible spending account