Lead Data Engineer - AWS, Databricks & ETL/ELT

Job Description

**FULLY REMOTE**

ScolerTec is seeking multiple Lead Data Engineers to support a large-scale cloud data modernization and governance program. This role will lead the design, development, testing, and maintenance of secure, scalable data engineering components and platform services supporting operational data, reporting, analytics, and AI/ML use cases. The Lead Data Engineer will work closely with architecture, governance, migration, and DevSecOps teams to deliver high-quality, auditable, and performant cloud-based data solutions using AWS and Databricks .

Key Responsibilities

\n
• Lead development of scalable cloud-based data platforms supporting data lake, lakehouse, and data warehouse architectures.
\n
• Design and develop enterprise ETL/ELT and ingestion pipelines for batch and near-real-time workloads.
\n
• Build and optimize data engineering solutions using Databricks, Spark/PySpark, Delta Lake, Python, and SQL .
\n
• Implement secure and auditable integrations across legacy, cloud, and external systems.
\n
• Lead implementation of operational databases, document storage, schemas, APIs, and data services.
\n
• Provide technical direction and oversight to Data Engineers, including design and code reviews.
\n
• Implement Infrastructure as Code for database and platform provisioning and configuration.
\n
• Implement data quality checks, validation, error handling, metadata, lineage, and governance-aligned structures.
\n
• Optimize pipeline performance, Spark workloads, query performance, scalability, and cloud cost.
\n
• Integrate pipelines with Government-provided CI/CD processes.
\n
• Support incident resolution, RCA, migration, archival, failover testing, and DR activities.
\n
• Maintain technical documentation, architecture artifacts, runbooks, and audit-ready deliverables.
\n

Qualifications

\n
• 7+ years of specialized data engineering experience, including leadership of enterprise-scale engineering efforts.
\n
• Strong hands-on experience with AWS data and compute services .
\n
• Strong hands-on production experience with Databricks, Spark/PySpark, and Delta Lake/lakehouse architectures .
\n
• Strong proficiency in SQL, Python, ETL/ELT, and data transformation .
\n
• Hands-on experience building enterprise-scale batch and near-real-time data pipelines.
\n
• Experience with AWS data services such as S3, Glue, Redshift, Kinesis, Lambda, Athena, or related services .
\n
• Experience with orchestration tools and streaming/near-real-time ingestion.
\n
• Experience with CI/CD, secure software engineering, and Infrastructure as Code.
\n
• Experience with data governance, metadata, lineage, data quality, and classification standards.
\n
• Experience leading and mentoring Data Engineers.
\n
• Federal or regulated-environment experience preferred.
\n

Preferred

\n
• AWS GovCloud experience.
\n
• Databricks experience with Unity Catalog, Workflows, Auto Loader, SQL Warehouses, Delta Live Tables/Lakeflow, or governed Gold-layer datasets .
\n
• Informatica or another enterprise integration/ETL platform.
\n
• Terraform or equivalent Infrastructure as Code.
\n
• Kafka or Kinesis streaming experience.
\n

Preferred Certifications

\n
• AWS Certified Data Engineer
\n
• AWS Certified Solutions Architect – Professional
\n
• AWS Certified Data Analytics – Specialty
\n
• Databricks Certified Data Engineer Professional
\n
• CDMP
\n