Senior Data Engineer

Job Description

Senior Data Engineer – PySpark & Databricks

Location: Charlotte, North Carolina (Hybrid)// Need only locals

Experience: 10+ Years

Job Type: Contract

Job Summary

We are seeking an experienced Senior Data Engineer with strong hands-on expertise in PySpark, Databricks, Python, and SQL to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have extensive experience building data pipelines, processing large datasets, developing ETL/ELT workflows, and optimizing data processing performance.

The candidate must possess strong programming and problem-solving skills, with the ability to write clean, efficient, and production-ready code.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark and Databricks.
• Write, test, debug, and optimize Python, PySpark, and SQL code.
• Build and maintain ETL/ELT processes for structured and unstructured data.
• Develop data transformations and workflows to process large volumes of data.
• Work with Databricks notebooks, Delta Lake, and Spark-based data processing frameworks.
• Optimize Spark jobs, troubleshoot performance issues, and improve pipeline efficiency.
• Implement data quality checks, validation, error handling, and monitoring.
• Integrate data from multiple sources, including databases, APIs, cloud storage, and enterprise systems.
• Collaborate with data architects, analysts, and business stakeholders to deliver reliable data solutions.
• Follow coding standards, version control practices, testing procedures, and CI/CD processes.

Required Skills

• 10+ years of experience in data engineering or related software/data development roles.
• Strong hands-on experience with Python and PySpark .
• Practical experience developing data solutions using Databricks .
• Strong SQL skills, including complex queries, joins, aggregations, and query optimization.
• Experience building and maintaining ETL/ELT data pipelines.
• Knowledge of Apache Spark architecture, distributed processing, and performance tuning.
• Experience with Delta Lake and lakehouse architecture.
• Strong coding, debugging, troubleshooting, and problem-solving skills.
• Experience with Git and software development best practices.
• Familiarity with cloud platforms such as AWS, Azure, or GCP.
• Strong communication and collaboration skills.

Preferred Qualifications

• Experience with Azure Databricks, AWS Databricks, or another cloud-based data platform.
• Knowledge of data warehousing, dimensional modeling, and data lake architecture.
• Experience with orchestration tools such as Airflow, Azure Data Factory, or Databricks Workflows.
• Familiarity with CI/CD, automated testing, and deployment practices.
• Experience working in Agile development environments.