Senior Data Platform Engineer

Hollstadt Consulting is a management and technology consulting firm dedicated to placing professionals at engagements where they will excel. When you work with us, you'll work with a refreshingly real company led and staffed by seasoned experts who are also down-to-earth, good people. We're committed to treating you with respect and helping you achieve your career aspirations. Since 1990, Hollstadt has been a trusted partner to more than 150 domestic and global companies and has successfully completed over 3,000 projects. Our continued growth has created challenging and rewarding opportunities for accomplished IT and Business Consultants. Hollstadt Consulting is an equal opportunity employer including disability/veteran. By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Hollstadt Consulting and its affiliates, and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel at any time. Job Description Role: Senior Data Platform Engineer Rate: $ 65.34/hour W2 Location: Minneapolis, MN - 3 days onsite required About the Platform The client's Data Analytics Platform (DAP) spans the full data lifecycle on Azure. Batch and file-based sources flow through a multi-zone Delta Lake medallion lakehouse on ADLS Gen2, processed with Azure Synapse Analytics. A real-time tier on Azure Data Explorer handles streaming and sensor telemetry. Azure SQL Database holds orchestration metadata and operational state, master data management provides enterprise golden records, containerized Python jobs on Azure Container Apps handle API extraction, and Logic Apps handle lightweight intake and notification workflows. Power BI delivers reporting tiered by data velocity: high-velocity real-time dashboards from Azure Data Explorer, near-real-time views from Azure SQL, and analytical reporting from curated serverless SQL marts in the lakehouse. Access is governed by a domain-based classification model aligned to Minnesota government data practices requirements, and all code deploys through Azure DevOps CI/CD. About the Role This is a generalist, delivery-focused engineering role built for independent work inside an established platform. The role reports administratively to an IT manager. The client's Data Architect owns the platform's architecture, standards, and conventions and reviews and approves design and code; within those guardrails, the engineer owns how the work gets done. The work is broad: onboarding new data sources, extending and hardening existing pipelines, resolving defects across the estate, building curated data products, and improving the shared framework itself where it falls short. We are not looking for a narrow specialist. We are looking for a specific working style: an engineer who can take an ambiguous request, investigate the systems involved, read the existing code and metadata before writing anything new, propose an approach, and carry the work to done. Direction comes from the platform's architecture and standards rather than step-by-step instruction: the Data Architect reviews and approves outcomes, and the engineer owns everything in between. You will succeed in this role if you Treat an unfamiliar system, API, or codebase as something to investigate, not a blocker. You read documentation, trace existing pipelines, test assumptions, and come back with findings and a proposed path, not just questions. Work within an established framework and its conventions, and improve it through its own patterns when it falls short, rather than building one-off solutions around it. Leave code better than you found it, with the discipline to keep changes surgical: fix what is broken, flag what is fragile, and avoid both unrequested rewrites and shortcuts that leave technical debt behind. Communicate status honestly and early, including when something is harder than expected or when you find a problem nobody asked about. Key Responsibilities Source onboarding: Build ingestion for new data sources (REST APIs, SFTP drops, network file shares, vendor exports, sensor and telemetry feeds) through the metadata-driven orchestration framework, landing raw data and promoting it through the medallion zones with PySpark notebooks and Synapse pipelines. Curated data products: Develop serverless SQL views and Delta Lake tables in the curated consumption layer that apply business rules and serve analytics, reporting, and downstream feeds, following the platform's schema-per-mart and access-control conventions. Extraction jobs: Develop and maintain containerized Python jobs on Azure Container Apps for API extraction workloads, including managed identity authentication, structured logging to the orchestration database, and YAML-based CI/CD deployment. Troubleshooting and defect resolution: Diagnose and resolve pipeline failures, data quality issues, and performance problems across the platform, tracing issues through orchestration metadata, Spark logs, and source systems to root cause. Framework improvement: Extend and refactor shared toolbox notebooks and framework components when the backlog exposes gaps, with regression awareness across all sources that depend on them. Metadata and documentation: Register new sources and datasets in platform metadata and document solutions for team knowledge sharing. Required Qualifications What we mean by experience: every item below requires sustained, hands-on production work. That is code the candidate personally designed, built, deployed, and supported in a live environment over time, not coursework, certifications alone, proof-of-concept exposure, or brief contact with a technology on someone else's project. Evaluation will include a detailed technical discussion of production work the candidate has personally delivered (design decisions, failure modes encountered, and how solutions held up under real data and real change) and a hands-on technical exercise using materials the client provides. Candidates will not be asked to show code or other proprietary artifacts belonging to prior employers or clients. 5+ years of data engineering experience with your own code running in production, including substantial hands-on work in Azure. Production proficiency with Azure Synapse Analytics: Spark notebooks, pipeline development, triggers, and serverless SQL pools that you have built and operated in a live environment. Note that Databricks and Microsoft Fabric are not used in this environment; native Synapse experience matters. PySpark and T-SQL at the depth that comes from regular production use: performance tuning, window functions, and incremental load patterns against Delta Lake tables on ADLS Gen2, with the ability to explain the choices behind your own code. Production-grade general-purpose Python beyond Spark: API clients, authentication handling, parsing varied file formats, and code you have packaged and deployed for containerized execution. Day-to-day working use of Azure DevOps (Git, YAML pipelines, pull-request workflow) and of managed identity and Entra ID authentication patterns for Azure services. A demonstrated record of self-directed delivery: named, specific examples of taking on unfamiliar systems or vaguely specified problems and driving them to a working, reviewed solution in production. Clear written and verbal communication, including the ability to document technical work for a small team. Preferred Qualifications Unlike the required items above, familiarity or lighter exposure is acceptable here. Experience with Azure Data Explorer (KQL, ingestion mappings, continuous export) or other real-time analytics stores. Experience working in a metadata-driven ingestion framework, where pipeline behavior is configured in database metadata rather than hard-coded per source. Experience with Azure Container Apps or Docker-based job workloads, Logic Apps, and infrastructure-as-code (Bicep). Familiarity with Power BI as a consumer of the platform, including DirectQuery and Government Community Cloud constraints. Microsoft certifications such as Azure Data Engineer Associate or Azure Solutions Architect. Prior work in government, transportation, aviation, utilities, or other regulated environments handling non-public data. Hollstadt offers medical, dental, vision, life insurance, short-term disability, long-term disability, paid sick leave, and retirement benefits to eligible employees. With three different medical plans to choose from, you can enroll in the coverage you need from individual to family, or anywhere in between! Remarketing Process Hollstadt is based on retention and relationships. We get to know your strengths and career wishes throughout your assignment and then start remarket discussions 6-8 weeks prior to your end date. By being proactive, we are able to keep your down time between assignments as short as possible, unless you choose otherwise. Professional Development Hollstadt offers on-demand training through our consultant portal. Trainings give our consultants the continuing education they need to excel on their projects. Many of our courses apply towards continuing education credits and we have an entire training hub dedicated to upskilling in Artificial Intelligence (AI). 401k + Matching One popular benefit is our 401(k) match on the first 4% of your contributions. Hollstadt wants to help you reach your long-term financial goals and understands that planning for your future is critical. Consultants also have access to support from a Financial Advisor. We appreciate and reward loyalty. Join Hollstadt, stay for 5 years, and we’ll give you a $5,000 Longevity Award bonus! Additionally, we know great talent knows other great talent. If you are on contract with Hollstadt and refer one of your connections who gets placed, we’ll pay you $1,000! Ongoing Support & Networking We have made a significant investment in building a support program for our consultant team - so you never have to feel like you are going it alone. We also have a Consultant Coach program which acts like a 'work buddy' to provide a safe ear for questions or concerns at your client site. #J-18808-Ljbffr Hollstadt & Associates