Analyst, Lead Data Engineering

DescriptionEnterprise Products Partners L.P. is one of the largest publicly traded partnerships and a leading North American provider of midstream energy services to producers and consumers of natural gas, NGLs, crude oil, refined products and petrochemicals. Our services include: natural gas gathering, treating, processing, transportation and storage; NGL transportation, fractionation, storage and import and export terminals; crude oil gathering, transportation, storage and terminals; petrochemical and refined products transportation, storage and terminals; and a marine transportation business that operates primarily on the United States inland and Intracoastal Waterway systems. The partnership’s assets include approximately 50,000 miles of pipelines; 260 million barrels of storage capacity for NGLs, crude oil, refined products and petrochemicals; and 14 billion cubic feet of natural gas storage capacity.We are currently seeking an experienced Data Engineer to join the AI and Advanced Analytics department. As part of the AI Engineering team, the Lead Data Engineer will architect, design, and implement a cloud based modern enterprise Data & AI platform to support AI and analytic use cases for the midstream oil and gas business units. This individual will shape the architecture, set engineering standards, and write production code across data products, AI models, and platform infrastructure.Responsibilities include, but are not limited to:Design and build reliable batch and streaming pipelines, including ingestion of industrial time-series and operational data alongside unstructured document sourcesEstablish data modeling, quality, lineage, and cataloging practices for data product developmentBuild feature pipelines and curated datasets that serve analytics, ML, and GenAI use casesProductionize classical ML models following MLOps practicesBuild GenAI applications such as retrieval-augmented generation over enterprise documents, including chunking, embeddings, vector search, prompt management, and evaluationImplement guardrails, observability, and cost controls for LLM-based systemsDefine evaluation approaches that make AI quality measurable and regressions visibleBuild self-service tooling, reusable templates, and CI/CD pipelines so teams can ship data and AI products consistently on AWS and on-premises environmentsOwn infrastructure as code, observability, and reliability practices for the platformEmbed security, governance, and compliance requirements into the platform by designDefine and implement best practices around AI Assisted Software Development Life CycleMentor junior engineers in design and code reviews and implementation best practicesQualificationsThe successful candidate will meet the following qualifications:10+ years of software engineering experience, with at least 2 years in a technical lead roleExpert level experience in Python with solid software engineering fundamentals including testing, code reviews, branching policies, version control, design patterns, and API designExperience building data products leveraging Snowflake, DBT, and AirflowExperience delivering classical ML and GenAI systems to production, including MLOps practices such as model registries, pipeline automation, and monitoringDeep AWS experience building data and AI platforms including S3 Tables, Glue, Athena, Lake Formation, SageMaker, Bedrock, Lambda, ECS, RDS, IAM, and VPCExperience in designing and implementing Infrastructure as Code (IaC) using Terraform, AWS CDK, or CloudFormation along with deployment automation with CI/CD pipelinesPractical experience with AI-assisted software development workflows using tools such as Claude Code or Cursor to design, implement, test, and refactor codeExperience in structuring work for AI coding agents such as writing clear specifications, providing context, breaking tasks down, and reviewing generated output criticallyClear communication skills and the ability to work with both technical and non-technical stakeholdersJob Field: Information TechnologyOrganization: Info TechShift: Day JobJob Level: Individual ContributorTravel: Yes, 10 % of the Time