Lead Data Engineer

Lead Data EngineerLocation: 100% Remote (Preferred: Chicago, IL / Chicagoland Area) Client Location: Chicago, IL Duration: 6+ Months Contract (Contract-to-Hire) Start Date: 07/20/2026 End Date: 12/31/2026 Position Type: Contract (C2C) No. of Positions: 1Position Overview: We are seeking a highly skilled Lead Data Engineer to join a growing Data & Analytics team responsible for building and modernizing an enterprise-scale AWS + Databricks Lakehouse platform. This is a hands-on technical leadership role where you will architect, develop, and optimize scalable data solutions that power analytics, reporting, and AI initiatives across the organization. The ideal candidate has extensive experience designing enterprise data platforms, building robust ETL/ELT pipelines, and working with cloud-native technologies such as Databricks, Apache Spark, Delta Lake, Unity Catalog, AWS, SQL, and Python. This position offers the opportunity to influence enterprise data architecture while mentoring engineering teams and collaborating closely with business stakeholders.Key ResponsibilitiesDesign, develop, and maintain scalable enterprise data pipelines using Databricks and AWS.Lead the architecture and implementation of modern Lakehouse solutions.Build high-performance ETL/ELT pipelines supporting both batch and streaming workloads.Develop data ingestion, transformation, and orchestration frameworks.Optimize Spark workloads for performance, scalability, and cost efficiency.Implement Delta Lake best practices for reliable and governed data management.Design and maintain enterprise data models supporting analytics and reporting.Collaborate with business stakeholders to translate requirements into scalable technical solutions.Establish engineering standards, coding best practices, and data governance processes.Monitor pipeline performance and troubleshoot production issues.Mentor junior engineers while remaining actively involved in hands-on development.Support CI/CD automation and DevOps best practices across the data platform.Required Technical SkillsDatabricksApache SparkDelta LakeUnity CatalogDatabricks WorkflowsLakehouse ArchitectureNotebook DevelopmentPerformance OptimizationAWS CloudAmazon S3IAMVPC & NetworkingCloud StorageSecurity & Access ManagementCloud-native ArchitectureProgrammingPythonSQLPySparkSpark SQLData Processing FrameworksData EngineeringETL / ELT DevelopmentData PipelinesData IntegrationData TransformationBatch ProcessingStreaming Data ProcessingDistributed ComputingData ArchitectureLakehouse ArchitectureMedallion ArchitectureData ModelingData WarehousingMetadata ManagementData GovernanceData QualityDevOps & AutomationCI/CD PipelinesGitAzure DevOps or GitHub ActionsInfrastructure AutomationMonitoring & LoggingRequired Qualifications10+ years of overall IT experience.5+ years of hands-on Data Engineering experience.Strong experience designing enterprise-scale ETL/ELT solutions.Expert-level experience with Databricks, Apache Spark, Delta Lake, and Unity Catalog.Strong AWS experience including S3, IAM, networking, and cloud architecture.Advanced SQL and Python programming skills.Experience building batch and streaming data pipelines.Strong understanding of distributed computing concepts.Experience implementing Lakehouse and Medallion architecture.Experience with Agile software development methodologies.Strong problem-solving and analytical skills.Excellent communication and stakeholder management abilities.Preferred QualificationsExperience leading Data Engineering teams.Experience supporting AI, Machine Learning, or Advanced Analytics platforms.Knowledge of enterprise data governance frameworks.Experience with infrastructure automation and cloud optimization.Exposure to modern analytics platforms and reporting ecosystems.Experience working in highly collaborative Agile environments.Soft SkillsStrong leadership and mentoring abilities.Excellent communication and presentation skills.Ability to translate business needs into technical solutions.Self-driven with strong ownership and accountability.Comfortable working in fast-paced, evolving environments.Strong organizational and time-management skills.Collaborative mindset with a focus on delivering business value.