Senior Data Engineer / Data Architect -- 13886

Job Description

Job Description

Senior Data Engineer / Data Architect

Our client, a growing mid-market company, is investing in a modern enterprise data platform built on Azure Databricks to support analytics, reporting, and emerging AI initiatives. This is a hybrid engineering and architecture role covering everything from pipeline builds to data modeling, governance, and platform optimization within a Lakehouse framework.

Responsibilities

• Develop batch and streaming pipelines in Databricks and Spark, moving data from source to consumption
• Build out the medallion (Bronze/Silver/Gold) layer structure within the Lakehouse
• Bring data in from ERP, SaaS tools, APIs, and older legacy platforms
• Improve pipeline performance while keeping an eye on cost and scale
• Set up CI/CD workflows through GitHub
• Take ownership of data architecture direction and set the standards other teams build against
• Create logical and physical data models spanning finance, operations, and field service
• Lock in consistent definitions for shared entities like customer, job, and revenue
• Shape data structures so they work equally well for BI reporting and AI/ML training
• Run governance through Unity Catalog, covering access roles, lineage, and audit needs
• Put standards in place for naming, permissions, and data quality checks
• Keep HR and financial data protected in accordance with security policy
• Build checks that catch data quality issues before they reach reporting or downstream systems
• Design repeatable patterns for how data flows in, gets transformed, and gets consumed by BI tools, APIs, or AI models
• Link Databricks into the broader Azure ecosystem (ADLS, internal APIs, enterprise apps)
• Keep the platform observable, stable, and running smoothly
• Get data ready for AI/ML consumption, including automation-focused use cases
• Collaborate with analytics and AI groups on datasets built for predictive work
• Sit with business and application teams to turn their needs into working data solutions
• Set direction on engineering approach and architectural tradeoffs for the broader team
• Coach junior team members and contractors as the group expands

Requirements

• 6+ years building data pipelines, data architecture, or enterprise data platforms
• Direct hands-on background standing up cloud-based data pipelines
• Prior work inside enterprise systems (ERP, CRM, or comparable platforms)
• Strong Azure Databricks and Apache Spark chops
• Solid Python/PySpark skills
• Advanced SQL
• Comfortable with Azure Data Lake Storage (ADLS)
• CI/CD background using GitHub
• Familiarity with API-driven and data integration patterns
• Comfortable with both dimensional and normalized modeling approaches
• Working knowledge of data governance and security, including access roles and lineage tracking
• Delta Lake or broader Lakehouse experience is a plus
• Any exposure to ML/AI pipelines or data science work is a plus
• Manufacturing, oil & gas, or ERP-heavy backgrounds are a plus
• Experience with C#, .NET, SQL, or JavaScript is a plus
• DevOps and source control experience tied to ERP development is a plus
• Understanding of cloud ERP platforms and migration paths is a plus
• Power BI, SSRS, or Tableau exposure for integration/reporting is a plus

Do not apply unless you are authorized to work in the United States for any employer as client company cannot sponsor or transfer visas at this time.

About Walker Elliott

We connect, advise and support companies and new employees to help them grow and thrive. As a mid-size recruitment firm, we focus on the specialized sectors of upstream oil and gas and information technology. With us, you get the best of both worlds - local expertise together with the finest national market intelligence needed to find great candidates. Our people have an average tenure of more than ten years in their specific area of technical expertise, and our hands-on service approach is methodical, personalized and proven. We know that hiring people with the right attitude, behaviors and cultural fit is critical to ensuring high performers for your business. That’s why our skilled recruiters work closely as a team and combine their unique expertise, knowledge, and skills to deliver the best results.

Company Description

About Walker Elliott\r\n\r\nWe connect, advise and support companies and new employees to help them grow and thrive. As a mid-size recruitment firm, we focus on the specialized sectors of upstream oil and gas and information technology. With us, you get the best of both worlds - local expertise together with the finest national market intelligence needed to find great candidates. Our people have an average tenure of more than ten years in their specific area of technical expertise, and our hands-on service approach is methodical, personalized and proven. We know that hiring people with the right attitude, behaviors and cultural fit is critical to ensuring high performers for your business. That’s why our skilled recruiters work closely as a team and combine their unique expertise, knowledge, and skills to deliver the best results.