Lead Data Engineer

Job Description

Job Description

Role: Lead Data Engineer
Location: 77380
Duration: 6 Month Contract-to-hire
Work Authorization: US Citizens and Green Card Holders ONLY. This role will have access to federal government information. C2C and third-party candidates are ineligible. 

Job Summary:
We are seeking a highly motivated Lead Data Engineer with a passion for data modeling, modern data architecture, and cloud-native engineering practices. This role is responsible for leading the technical design and implementation of enterprise data platforms while remaining hands-on in development.

The ideal candidate will own solution architecture, technical implementation, delivery planning, estimations, and technical leadership while mentoring engineering team members. This individual will design scalable, reliable, and high-performance data platforms using a Databricks-native architecture built on Delta Lake, Unity Catalog, Delta Live Tables (DLT/Lakeflow Declarative Pipelines), and DBT.

This position partners closely with Product, QA, Project Management, and Business stakeholders to deliver trusted, analytics-ready data products while proactively managing technical risks, dependencies, and delivery timelines.

Duties/Responsibilities:
Lead Data Engineering & Technical Delivery

• Own the end-to-end technical data engineering for enterprise data platforms.
• Lead technical implementation while remaining hands-on with development.
• Provide delivery estimates, sprint planning input, and technical guidance to engineering teams.
• Mentor and coach data engineers while establishing engineering best practices.

 
Design & Implement Data Models

• Design and maintain enterprise dimensional data models that support scalable reporting and analytics.
• Optimize Gold-layer Delta tables and dimensional models to minimize downstream Power BI DAX complexity through upstream data transformations.
• Apply best practices in data warehousing, semantic modeling, and modern lakehouse architecture.

 
Build Modern Databricks Pipelines

• Design and develop modular, reusable, metadata-driven ELT pipelines using Databricks, PySpark, Delta Live Tables (Lakeflow Declarative Pipelines), Unity Catalog, and DBT.
• Implement scalable orchestration patterns using Databricks-native architecture.
• Build robust, maintainable data pipelines following engineering best practices.

 
Develop Data Transformations

• Build high-performance transformations using SQL, PySpark, and DBT.
• Implement data quality validations, schema evolution strategies, and automated lineage.
• Develop scalable transformation frameworks that support reusable engineering patterns.

 
Build Metadata-Driven Frameworks

• Design and implement metadata-driven frameworks, code generation solutions, and agentic engineering patterns to improve engineering productivity and standardization.
• Promote reusable architecture patterns across data engineering initiatives.

 
Enable Quality Engineering

• Partner closely with QA teams to define testing strategies, validation criteria, and automated testing for data accuracy, completeness, and reliability.
• Ensure robust quality controls are incorporated throughout the engineering lifecycle.

 
Delivery Leadership

• Proactively communicate technical progress, sprint status, delivery timelines, project dependencies, and technical risks to Project Management.
• Raise technical blockers early, escalate issues appropriately, and proactively identify delivery risks without requiring micromanagement.

 
Enable CI/CD & DevOps

• Integrate data engineering workflows with Git and Azure DevOps for source control, automated testing, and continuous deployment.

 
Support Analytics

• Collaborate with analytics, BI, and business stakeholders to ensure data models are optimized for reporting and self-service analytics.

 
Troubleshoot & Optimize

• Continuously optimize pipeline performance, storage efficiency, and query execution.
• Monitor production environments and ensure high availability, reliability, and scalability.

 
Continuous Improvement

• Stay current on Databricks, Delta Lake, DBT, and modern data engineering best practices, driving continuous improvements across the platform.

Required Skills/Abilities:

• 10+ years of overall experience in data engineering.
• 5+ years of hands-on Databricks experience designing and delivering enterprise data platforms.
• Demonstrated experience serving as a technical lead responsible for solution architecture, implementation, estimations, delivery planning, and mentoring engineering teams.
• Expert-level experience with Databricks, Delta Lake, Unity Catalog, Delta Live Tables (DLT/Lakeflow Declarative Pipelines), and DBT.
• Strong SQL and PySpark development skills.
• Deep experience developing scalable DBT models and transformation frameworks.
• Proven expertise in dimensional modeling, semantic modeling, and enterprise data warehouse design.
• Deep experience optimizing Gold-layer Delta tables and dimensional models to simplify downstream Power BI DAX calculations through upstream transformations.
• Experience designing metadata-driven frameworks, code generation solutions, or agentic engineering patterns.
• Experience implementing enterprise data quality frameworks and partnering with QA teams to define testing strategies and validation criteria.
• Experience proactively communicating technical blockers, project dependencies, sprint progress, delivery timelines, and technical risks to Project Management.
• Strong understanding of Delta Lake architecture, performance optimization, and modern Lakehouse design principles.
• Experience with Git-based source control, CI/CD, and Azure DevOps.
• Excellent analytical, communication, leadership, and problem-solving skills.

 
Education and Experience

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent practical experience).
• Minimum of 10 years of professional data engineering experience.
• Minimum of 5 years of hands-on Databricks development experience in enterprise environments.

Our benefits include:

• Comprehensive medical benefits
• Competitive pay
• 401(k) retirement plan
• ...and much more!

About INSPYR Solutions
Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.