Principal, Data Scientist, Agentic AI Systems Engineering & Model Post-Training

Reference: R-2612602Posted: 2026-08-22Location: BENTONVILLE, AR, United StatesSalary: $110000 - $220000/yearType: Full time / Regular/PermanentCompany: WalmartBusiness Segment: Home OfficeThe Opportunity: Walmart’s Supply Chain AI Lab & Innovation Factory is building a new generation of production-grade agentic AI systems that reason over complex enterprise information, coordinate specialized agents, use tools safely, plan and execute long-horizon work, and continuously improve through rigorous evaluation and model post-training. This role exists to build those systems end to end—and to improve the models that power them. This is not an analytics-focused data science role. It is a deeply hands-on AI systems engineering position focused on designing, building, and operating production software. As a Principal Data Scientist in this space, you are a hands-on technical leader. You quickly turn hard, ambiguous problems into working full-stack prototypes—with a real user experience, APIs, telemetry, and an evaluation plan—then harden them into secure, reliable, observable, maintainable production systems. You independently own major product and platform domains within the shared agentic architecture, across the full technology stack—agent orchestration and model logic, backend services and APIs, the data layer, and web and CLI/TUI interfaces. What you will build and own:Build advanced agentic systems end to end:Design, build, test, launch, and operate production agentic applications and services with multi-step and long-horizon planning, tool use, retrieval, durable sessions and workflows, context management, multi-agent orchestration, human approval, and safe recovery when decisions or actions need intervention. Build a policy-first agent runtime and control plane with deterministic allow, deny, and ask decisions; least-privilege tool and data access; scoped identity and authorization; auditable human approvals; bounded subagent delegation; and safe cancellation, user steering, and retry behavior for long-running work. Build and own the full product path as a full-stack systems engineer: backend services and APIs, the data layer (database and schema design, data modeling, migrations, and streaming pipelines), telemetry, and accessible React/TypeScript experiences for associates and operators. Where the workflow demands it, design equally usable AI-native CLI/TUI or headless, structured-output interfaces that support automation, operations, and CI/CD-style integration. Where you will apply this: These capabilities will first be applied to some of Walmart's most complex operational and supply-chain problems, beginning with the Autonomous Supply Chain Engine and its Discovery Loop—a continuous, human-in-the-loop mechanism that monitors signals, discovers opportunities and risks, reasons through changing conditions, and generates strategic recommendations. Build and own key capabilities of the Discovery Loop within our Autonomous Supply Chain Engine. This is a flagship capability for this role: you will own major parts of its execution model, knowledge and context layer, evaluation loop, and learning path that turns feedback and business outcomes into measurable improvement. Prior supply chain or logistics experience is not required. You bring outstanding depth in agentic AI engineering, software engineering, and machine learning engineering, and you will work directly with AI, product, engineering, operations, data, and strategy partners on high-priority work. Engineer the agentic AI foundation:Own the design and build of advanced multi-agent harnesses, runtimes, and orchestration capabilities using frameworks such as Pydantic AI, LangGraph, LangChain, AutoGen, or LlamaIndex, or purpose-built custom infrastructure. Building your own runtime where that is the right call is a strength, not a gap. Establish typed contracts, testability, scalability, operability, and developer ergonomics as non-negotiable platform properties. Engineer explicit, inspectable agent execution. Design graph-based, state-machine-based, event-driven, planner/executor, or equivalent execution models as the problem warrants, with explicit state, typed dependencies and handoffs, conditional branching, fan-out and fan-in to subagents, retry and repair paths, loop detection, cost and time budgets, and hard termination conditions. Design external verification into the system—test runners, execution results, transaction outcomes, deterministic checks, and expert human review—because a model reviewing its own output is not verification. Build governed knowledge and context infrastructure for durable agent memory. Choose and combine the right representations—knowledge and context graphs, vector retrieval, relational and temporal models, or hybrids—with an explicit schema and ontology treated as a product contract. Own entity resolution and conflict handling, construction and enrichment pipelines from structured and unstructured sources, provenance and temporal validity, schema validation and evolution, and multi-hop retrieval that answers questions flat retrieval cannot. Build and operate agent skills, tool adapters, hooks and extension points, Model Context Protocol (MCP) clients and servers, structured outputs, function calling, enterprise APIs, identity, and secrets management. Own MCP lifecycle and reliability end to end: secure configuration and authentication, per-agent tool binding, schema compatibility, tool discovery, timeouts, health monitoring, retries, circuit breaking, quarantine, cleanup, failure isolation, and auditable operations. Create governed extension ecosystems for agents, skills, commands, plugins, hooks, tool adapters, and reusable workflows. Define stable contracts, compatibility and versioning strategies, secure installation and update paths, isolation boundaries, rollback behavior, and observability so extensibility does not become an uncontrolled code-execution surface. Treat agent quality as an engineering discipline: golden tasks, offline benchmarks, online experiments, adversarial and regression testing, failure analysis, measurable quality thresholds, and release gates. Establish practical AgentOps / LLMOps practices for prompt and tool versioning, tracing, evaluation datasets, workflow reliability, cost and latency controls, incident learning, and continuous improvement of long-running autonomous systems. Instrument the system so engineers and operators can answer, with evidence, what the agent did, why it was permitted, which model, tool, and policy version was involved, what data and integrations were used, what it cost, where it failed, and how to reproduce or remediate the outcome safely. Advance the models themselves Improve model reasoning and quality through hands-on post-training for the systems you own: reinforcement learning (RLHF/RLAIF), preference optimization, supervised fine-tuning, distillation, and reward and grader design to strengthen reasoning, tool-use reliability, and domain-specialized behavior across frontier models and smaller, domain-specialized models. Build the model-improvement flywheel. Turn production interaction traces, tool-use trajectories, human feedback, and successful and failed reasoning paths—together with Walmart's proprietary enterprise and operational data, synthetic data, and curated evaluation sets—into governed training and evaluation datasets. Use them to post-train, distill, evaluate, and redeploy increasingly capable domain-specialized models back into the agentic systems, so the platform you build continuously improves the models that power it. Own the data governance, provenance, privacy, and access controls that make this safe at enterprise scale. Design provider-aware, model-agnostic execution and routing layers that account for model capabilities—including multimodal inputs and outputs across text, images, and documents—context limits, structured outputs, streaming behavior, credentials, rate limits, transient failures, and explicit quality, latency, and cost trade-offs. Make every solution safe, scalable, and production-ready Use Google Cloud Platform (GCP) or comparable cloud platforms together with Walmart internal technologies to build systems that are secure, fault tolerant, cost-aware, observable, highly available, and low latency. Build responsible enterprise-agent behavior through least-privilege access, explicit authorization boundaries, complete audit trails, data-protection controls, defenses against prompt injection and untrusted instructions, approval controls for consequential actions, and reversible recovery paths. Design distributed data, streaming, and telemetry systems using technologies such as Kafka, Flink, Spark, OpenTelemetry, and Grafana; use operational signals to improve product quality, reliability, and user trust. Set quality, performance, and release standards with layered unit, integration, API, workflow, and user-journey testing; adversarial safety testing; benchmark- and profile-driven performance work; CI/CD quality gates; reliable rollout and rollback practices; and incident-response readiness. Deliver accessible associate-facing experiences that meet applicable Walmart accessibility standards, including WCAG 2.2 AA where applicable. Raise the bar around you Mentor senior engineers through code, design, operational leadership, and clear technical judgment while staying immersed in implementation and customer and associate outcomes. Raise the engineering bar through clear technical writing, rigorous code and architecture reviews, and reusable platform capabilities other teams can build on. Make consequential technical trade-offs based on evidence, security, maintainability, measurable outcomes, and user needs—not novelty for novelty's sake. Move quickly without cutting corners: create evidence-driven prototypes, validate them with real users and measurable evaluations, then harden successful ideas into production-grade systems. The bar for this role We are hiring for impact that extends beyond your own scope of work. Delivering strongly against your own responsibilities is the starting point for this position, not the measure of success. At the Principal level we expect you to make the teams and products around you measurably better. In practice, that means raising the technical performance of the engineers you work with, creating new capability rather than only consuming what already exists, taking ownership of difficult and high-consequence problems that others step around, and building foundations that keep paying off after the project that created them is finished. What you will bring 8+ years of hands-on software, machine learning, or applied AI engineering experience with Staff/Principal-level technical impact across teams—capable of independently taking a frontier-grade, full-stack agentic product from prototype to production (agent runtime, CLI/TUI, web frontend, backend services and APIs, and the underlying data layer) while remaining deeply involved in code, design, delivery, and production operations. Co-equal depth in agentic AI engineering, software engineering, and machine learning engineering: ML and LLM fundamentals, experimental design, retrieval and ranking, model evaluation, production trade-offs, and sound judgment about when automation or an LLM is not appropriate. Strong, hands-on experience with model post-training—reinforcement learning (RLHF/RLAIF), preference optimization, supervised fine-tuning, distillation, and reward and grader design—applied to improve reasoning, tool-use reliability, or domain-specialized model behavior. Demonstrated experience shipping stateful, policy-controlled agent runtimes and tool-using multi-agent systems for autonomous long-horizon work: durable session lifecycle, context-window management, retrieval and RAG, structured outputs, guardrails, evaluation systems, secure delegation, safe tool use, interruption handling, and recovery—built on established orchestration frameworks or on custom infrastructure of your own design, including MCP and production tool and integration lifecycle management. Broad applied ML depth across several areas such as reinforcement learning, multimodal modeling, retrieval and ranking, forecasting, optimization and operations research, experimentation, or model evaluation—together with production ML lifecycle and MLOps practices, and judgment about when a classical model beats an LLM. Strong Python plus production backend capability in one or more of Go, Java, or TypeScript, with true full-stack capability (React/TypeScript frontend, APIs, and the data layer) that real operational users can trust. Strong production experience with distributed systems, GCP or comparable cloud platforms, Kubernetes/Docker, CI/CD, data and streaming systems, telemetry and observability, SLOs and SLIs, reliability practices, secure enterprise integrations, and layered testing across unit, integration, workflow, and user-journey behavior—plus the ability to learn and operate within Walmart's internal technology ecosystem quickly. Deep fluency with AI-native software-development workflows and coding agents such as Claude Code, Codex, Cursor, Copilot, or equivalent tools, applied with rigorous engineering judgment, code review, evaluation, and testing. Demonstrated evidence of exceptional hands-on depth and impact beyond your own scope. We accept many forms of proof: shipped systems you can speak to in depth, an architecture and code walkthrough, open-source contributions where disclosure is permitted, technical demonstrations, publications, or patents. We recognize that much frontier work is confidential; a public portfolio is welcome but is not required. Be prepared to describe work that measurably improved the teams and products around you. A highly motivated, proactive, ethical, and approachable builder with technical creativity and intellectual rigor, who challenges assumptions constructively, communicates clearly, and carries solutions from blank page to dependable production service. Minimum qualifications Applicants must meet one of the corporate eligibility paths below. The role-specific hiring bar above—including 8+ years of hands-on engineering experience—governs selection for this Principal position. Bachelor's degree in Computer Science, Computer Engineering, Computer Information Systems, Software Engineering, or a related area and 4 years of experience in software engineering or a related area; or 6 years of experience in software engineering or a related area. Preferred qualifications Master's or Ph.D. in Computer Science, Computer Engineering, Machine Learning, Artificial Intelligence, or a closely related Experience building developer-facing agentic platforms, SDKs, or coding and automation agents used by other engineers. Graph and knowledge-representation depth is a strong plus. Knowledge and context graphs, ontology and schema design, entity resolution, multi-hop retrieval, graph databases and query languages, or graph ML and network analysis. Experience with ML platform engineering, model serving, retrieval and ranking systems, experimentation platforms, and production LLM quality evaluation. Experience building trusted associate-facing products, complex operational systems, high-volume data and streaming platforms, or reusable AI foundations. Strong open-source contributions, technical talks, developer tools, or shipped agentic or ML products that demonstrate practical engineering depth. Evidence of building innovative technologies or applications that improve complex operational workflows for frontline users, associates, customers, or other high-impact users. At Walmart, we offer competitive pay as well as performance-based bonus awards and other great benefits for a happier mind, body, and wallet. Health benefits include medical, vision and dental coverage. Financial benefits include 401(k), stock purchase and company-paid life insurance. Paid time off benefits include PTO (including sick leave), parental leave, family care leave, bereavement, jury duty, and voting. Other benefits include short-term and long-term disability, company discounts, Military Leave Pay, adoption and surrogacy expense reimbursement, and more. You will also receive PTO and/or PPTO that can be used for vacation, sick leave, holidays, or other purposes. The amount you receive depends on your job classification and length of employment. It will meet or exceed the requirements of paid sick leave laws, where applicable. For information about PTO, see . Live Better U is a Walmart-paid education benefit program for full-time and part-time associates in Walmart and Sam's Club facilities. Programs range from high school completion to bachelor's degrees, including English Language Learning and short-form certificates. Tuition, books, and fees are completely paid for by Walmart. Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to a specific plan or program terms. For information about benefits and eligibility, see One.Walmart. The annual salary range for this position is $110,000.00 - $220,000.00 Additional compensation includes annual or quarterly performance bonuses. Additional compensation for certain positions may also include : - Stock