Lead Data Engineer

Lead Data EngineerLocation: Charlotte, NCDuration: 12 monthsPay Rate: $90 - $100/HR (W2 Only)Job/Role Description:Lead the design, architecture, and implementation of enterprise-scale data engineering solutions across AWS cloud environments.Design, develop, and optimize scalable, resilient data pipelines, ETL processes, data ingestion frameworks, and orchestration workflows.Architect and oversee enterprise data lake and data warehouse solutions using AWS Lake Formation, Amazon Redshift, Amazon Athena, S3, and related AWS technologies.Collaborate with Lead Developers, Data Scientists, Architects, Product Owners, and business stakeholders to define technical strategy and scalable data solutions.Provide hands-on technical leadership, engineering oversight, code reviews, and mentorship to Data Engineers and development teams.Drive architectural decisions in partnership with Data and Solution Architects to ensure scalability, security, reliability, performance, and maintainability.Design and support Kafka-based streaming and event-driven data architectures, preferably using Confluent Kafka.Develop distributed data processing solutions using Python, PySpark, AWS EMR, and other cloud-native technologies.Establish engineering standards and best practices for data modeling, ETL frameworks, pipeline reliability, monitoring, observability, and operational excellence.Lead end-to-end solution delivery while ensuring alignment with business requirements, enterprise architecture standards, security controls, and regulatory requirements.Design and implement Infrastructure as Code solutions using Terraform across multiple AWS accounts and environments.Develop and maintain CI/CD frameworks using GitHub and GitHub Actions.Oversee production support and operational management of AWS-based data platforms, including root-cause analysis, troubleshooting, and performance optimization.Champion data governance, metadata management, data quality, observability, and data stewardship practices across platforms and teams.Identify opportunities to modernize data architecture and improve operational efficiency through automation and cloud-native technologies.Support the development of AI-ready data pipelines and machine learning workflows, including feature engineering and MLOps.Design data solutions capable of supporting Generative AI applications, intelligent search, RAG architectures, LLMs, and vector databases.Drive technical decision-making and clearly communicate complex data architecture concepts to both technical and non-technical stakeholders.Required Qualifications:8+ years of Data Engineering experience, including at least 5+ years of extensive experience working within AWS environments.Expert-level experience with AWS services including S3, EMR, Glue Jobs, Lambda, Athena, CloudTrail, SNS, SQS, CloudWatch, and Step Functions.Strong hands-on experience designing and implementing enterprise-scale data lakes and data warehouses using AWS Lake Formation, Amazon Redshift, and Amazon Athena.Advanced Python development experience with extensive hands-on use of PySpark.Advanced SQL expertise, including query optimization, large-scale data processing, and enterprise data warehousing.Strong data modeling experience, including dimensional modeling, Data Vault, and other enterprise data modeling techniques.Deep experience designing, developing, and optimizing scalable and resilient data pipelines within AWS environments.Strong experience with distributed data processing frameworks, particularly PySpark and AWS EMR.Extensive knowledge of database management, performance tuning, and data architecture best practices.Experience with Kafka-based streaming architectures, preferably Confluent Kafka.Expertise with Infrastructure as Code using Terraform.Experience designing and implementing CI/CD frameworks using GitHub and GitHub Actions.Deep knowledge of AWS IAM roles, policies, governance, security controls, and cloud security best practices.Strong experience with workflow orchestration platforms such as AWS Step Functions, Apache Airflow, or equivalent technologies.Experience leading cloud migration, modernization, or enterprise data platform initiatives.Strong understanding of data governance, metadata management, data quality frameworks, observability, resiliency, and operational supportability.Experience creating AI applications using AWS Bedrock.Experience building AI-ready data pipelines and ML workflows, including feature engineering and MLOps.Knowledge of Generative AI technologies, LLMs, Retrieval-Augmented Generation (RAG), and vector databases.Experience working with cloud-based AI and data platforms such as AWS, Azure, or Databricks.Ability to lead hands-on development efforts while providing technical direction, engineering oversight, and code reviews across multiple initiatives.Experience establishing cloud development environments, infrastructure standards, security controls, and migration strategies across multiple accounts and environments.Proven ability to identify data gaps, develop strategic remediation plans, and implement scalable automation solutions.Experience designing highly reliable data pipelines with a strong emphasis on data quality, observability, resiliency, and operational support