Lead Data Engineer - Databricks, PySpark & AWS

Location: Houston, TX - The Galleria (100% Onsite) Experience: 8+ years Open to GC holders or US Citizens Overview We are seeking a Lead Databricks Engineer to support large-scale data platform initiatives focused on data modernization, cloud migration, and advanced analytics . This is a hands-on senior role requiring deep Databricks expertise, strong AWS experience, and the ability to partner closely with business and technical stakeholders. The ideal consultant has led complex Databricks implementations end-to-end, can code at a high level, and has experience modernizing legacy data platforms into scalable, cloud-native architectures. Key Responsibilities Lead the design and implementation of enterprise-scale Databricks solutions on AWS Drive data modernization initiatives, including lift-and-shift and re-architecture of legacy data platforms Build and optimize data pipelines using Python, PySpark, and SQL Design and manage Delta Lake architectures and implement Unity Catalog for governance and access control Develop and support streaming and batch data workloads Configure and optimize Databricks clusters and serverless compute for performance and cost Integrate Databricks with upstream and downstream systems (APIs, data sources, analytics tools) Partner with stakeholders to gather user and business requirements and translate them into technical solutions Implement observability, monitoring, and cost controls , including usage, volume, and pricing metrics Support data domains related to billing, accounting, pricing, and volume metrics Provide technical leadership, best practices, and mentorship to engineering teams Required Skills & Experience 8+ years of experience in data engineering , with recent hands-on focus on Databricks Strong experience deploying Databricks on AWS Advanced coding skills in Python, PySpark, and SQL Deep knowledge of Delta Lake , Unity Catalog , and Databricks workspace governance Experience with streaming data (Structured Streaming, Kafka, or similar) Strong understanding of Databricks cluster management, serverless compute, and performance tuning Experience integrating Databricks with enterprise systems and data sources Proven ability to work directly with business and technical stakeholders Experience supporting financial data domains (billing, accounting, pricing, usage metrics) is highly preferred Strong communication skills and ability to lead technical discussions Experience with AKS / Kubernetes environments Nice to Have Databricks or AWS certifications Consulting or contracting background in large enterprise environments Work Requirements 100% onsite role in Houston, TX Must be authorized to work in the United States #J-18808-Ljbffr SGI