Sr Data Engineer, Data Analytics & Intelligence, NA

Sr Data EngineerVantage Data Centers is seeking a Sr Data Engineer to help build, operate, and scale the governed data foundation for Operations, North America. This role is designed for an engineer who can independently deliver production-ready pipelines, curated datasets, semantic-model inputs, and AI-ready data products that support reporting, Executive reporting insight preparation, and the emerging AI Insight Solution.As part of the Data Engineering & Business Intelligence team, you will be responsible for delivering reliable data products that support analytics, AI data agents, operational intelligence, reporting, and an emerging AI-enabled platform. You will work closely with IT Global, solution build teams, business SMEs, and data governance partners to ensure data products are secure, reusable, explainable, and aligned with enterprise AI / Fabric direction.Success in this position requires comfort with ambiguity, strong execution discipline, and accountability for building trusted data assets that can be reused across analytics, operational intelligence, and AI-enabled use cases.Essential Job FunctionsDesign, build, and maintain reliable, scalable data pipelines using Python and PySpark on the Microsoft Azure data platform.Develop and operate batch and incremental data pipelines leveraging Azure Data Factory for orchestration and Azure Data Lake Storage Gen2 as the primary data store.Build and maintain curated lakehouse / gold-layer datasets and semantic-model inputs that support governed operational insights and AI-enabled consumption.Independently implement SQL- and Spark-based transformations to produce curated datasets that support enterprise reporting, analytics, AI-enabled insight preparation, and downstream consumption.Take ownership of assigned data pipelines and datasets, including monitoring, troubleshooting, performance optimization, documentation, and production support.Work with Azure Synapse, Microsoft Fabric / Lakehouse patterns where applicable, and related Azure analytics services to support analytical workloads and data consumption patterns.Prepare structured operational data for AI-enabled use cases by documenting business rules, source lineage, data reliability constraints, known quality limitations, and data dictionary definitions.Support source visibility, confidence context, and Data Reliability & Trust Indicator integration where applicable so downstream analytics and AI outputs can be understood and trusted.Contribute to ontology, taxonomy, semantic model, and data dictionary alignment needed to connect operational context, KPIs, incidents, work orders, and other enterprise data domains.Collaborate with business analysts, operations SMEs, data stewards, IT Global, and cross-functional stakeholders to translate requirements into practical, working data solutions.Apply established data governance, security, access-control, data classification, and engineering standards to ensure compliant, maintainable, and scalable solutions.Identify, document, and route data-quality issues to accountable owners, helping improve source correction rather than masking defects downstream.Participate in code reviews, technical discussions, sprint planning, and platform improvement initiatives as an active contributor.Proactively identify data quality issues, pipeline risks, platform dependencies, and improvement opportunities, and communicate them clearly in a fast-paced environment.DutiesDevelop and maintain PySpark notebooks and jobs to ingest, transform, validate, and curate data within the enterprise data platform.Build and modify Azure Data Factory pipelines for batch and incremental data ingestion.Implement Spark-based transformations that write curated datasets to Azure Data Lake Storage Gen2 and/or Fabric Lakehouse patterns using established folder structures, naming conventions, and governance standards.Create and maintain SQL views, tables, lakehouse objects, and semantic-model inputs to support analytics, operational intelligence, and AI-enabled consumption patterns.Prepare datasets for Fabric Data Agent / AI agent use cases by documenting business rules, joins, grain, quality limitations, source lineage, and operational definitions.Respond to pipeline failures, data validation issues, operational alerts, and data-quality escalations with clear root-cause analysis and practical remediation steps.Perform performance tuning of Spark jobs and SQL workloads, including partitioning, filtering, incremental logic, query optimization, and resource-aware design within established architectural patterns.Validate data outputs with business partners, operations SMEs, and data stewards, and address defects or discrepancies through documented correction paths.Support observability, logging, and auditability practices for data pipelines and AI-consumable datasets where applicable.Commit code using Git, follow branching standards, participate in pull request reviews, and support CI/CD ways of working using GitHub, Azure DevOps, or similar tools.Update documentation for pipelines, datasets, data contracts, data dictionaries, business rules, and operational runbooks as changes are made.Execute assigned backlog items within sprint timelines and raise risks, dependencies, or blockers early.Additional duties as assigned by management.Job RequirementsEducation & ExperienceBachelor's degree in Engineering, Computer Science, Data Analytics, or a related field, or equivalent experience.Minimum of 5–8 years of experience in data engineering, analytics engineering, or a closely related technical data role.Proficiency in Python for building and maintaining data pipelines, automation, data processing workflows, and PySpark-based transformations.Proficiency in SQL for querying, transformation, analytical data processing, model validation, and data quality checks.Solid understanding of ETL/ELT pipelines, data transformation patterns, data integration concepts, incremental processing, and production support practices.Experience analyzing enterprise data sources to identify data relationships, transformations, business rules, grain, ownership, and quality constraints.Experience building solutions on the Microsoft Azure platform with exposure to Azure Data Factory, Azure Synapse, Azure Data Lake Storage Gen2, Microsoft Fabric / Lakehouse patterns, and related analytics services.Working knowledge of data modeling fundamentals, including fact and dimension tables, semantic models, reusable data products, and analytics-ready structures.Experience supporting governed data products, including metadata, lineage, issue documentation, access-control awareness, data-quality validation, and operational runbooks.Experience working with source control and CI/CD workflows using tools such as GitHub or Azure DevOps.Strong communication and interpersonal skills with the ability to collaborate across IT Global, business SMEs, data governance partners, platform teams, and operations stakeholders in a fast-paced environment.Experience working in Agile development environments and using collaboration or project tracking tools such as Jira or similar tools.Travel required is expected to be up to 10% but may increase over time as the business evolves.Desired QualificationsExperience working with distributed data processing frameworks, including Apache Spark.Experience preparing governed data products for AI-enabled use cases, including Microsoft Fabric Lakehouse, semantic models, Fabric/Data Agent patterns, ontology or taxonomy alignment, and explainable AI outputs.Familiarity with data observability, metadata management, lineage, data contracts, reliability indicators, and operational best practices in production environments.Familiarity with additional Azure services such as Azure Functions or Logic Apps in support of data workflows.Experience supporting data platform enhancement, refactoring, modernization, or reusable architecture initiatives.Experience working with structured and unstructured operational sources, such as enterprise applications, operational workflows, documents, dashboards, and knowledge assets.Experience working in a scaling or fast-paced organization where priorities evolve quickly and practical delivery discipline is required.Physical Demands and Special RequirementsThe physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.While performing the duties of this job, the employee is occasionally required to stand; walk; sit; use hands to handle, or feel objects; reach with hands and arms; climb stairs; balance; stoop or kneel; talk and hear. The employee must occasionally lift and/or move up to 25 pounds.Additional DetailsSalary Range: $130k -155k (this range is based on Colorado market data and may vary in other locations)This position is eligible for company benefits including but not limited to medical, dental, and vision coverage, life and AD&D, short and long-term disability coverage, paid time off, employee assistance, participation in a 401k program that includes company match,