Data Analyst

Data Analyst Hybrid In Scottsdale, Az

The recently shared job description focuses on deterministic testing, data validation, and testing scenarios pertains to these additional openings. Role Overview The primary focus of these positions is validating AI-generated responses and ensuring data accuracy. Responsibilities include:

• Reviewing questions submitted by users.
• Tracing the underlying code and data sources used to generate responses.
• Verifying the accuracy of the logic, code, and resulting data.
• Testing and validating outputs to ensure quality and reliability.

The good candidates should possess:

• Advanced SQL expertise (required).
• Robust Python development skills (required).
• Experience with Visual Studio Code (VS Code) and related development environments.
• Robust analytical and critical-thinking abilities.
• The ability to understand user intent, not just the literal wording of a request, when validating responses.

Key responsibilities:

1. Deterministic Testing & Data Validation: Validate generative AI tool outputs for structured, rules-based use cases by reconciling results against trusted data sources and established SQL-based metrics. Ensure consistency, explainability, and auditability of outputs by confirming alignment with existing data pipelines and query logic. Expand and maintain test coverage across prioritized use cases to establish a robust, high-confidence baseline for the platform. Partner with data engineering and analytics teams to identify and resolve discrepancies in underlying data or logic.
2. Non-Deterministic Testing & Scenario Evaluation: Design and execute scenario-based testing for more complex, AI-driven outputs where direct validation is not always possible. Evaluate results based on intent accuracy, reasonableness, and confidence thresholds rather than exact match validation. Prioritize testing across higher-risk and high-impact use cases using curated question sets and real-world scenarios. Identify patterns in output variability and drive iterative refinement to improve reliability and user trust.
3. Human-in-the-Loop Review & Continuous Monitoring: Conduct ongoing review of generative AI tool interactions post-launch, validating outputs and ensuring quality across all user scenarios. Identify edge cases, inconsistencies, and emerging risks, and escalate findings to product and engineering teams. Synthesize insights from testing and live usage to inform enhancements, training data improvements, and governance practices. Serve as an accountable reviewer, providing a critical control point for responsible AI deployment and continuous improvement.

Required skills & experience: Robust SQL skills required. Robust analytical background with experience in data validation, SQL, and analytics workflows. Ability to assess outputs both quantitatively (data accuracy) and qualitatively (reasonableness, business context). Demonstrated critical thinking and sound judgment, especially in ambiguous or non-deterministic environments. Experience working with large datasets, reporting tools, or analytics platforms.

Preferred qualifications: Exposure to AI/ML or generative AI tools and associated testing or validation frameworks. Experience in scenario-based testing, UAT, or model validation. Familiarity with financial services, retirement, or plan sponsor analytics.