Senior Data Engineer

Senior Data Engineer – PySpark & DatabricksLocation: Charlotte, North Carolina (Hybrid)// Need only localsExperience: 10+ YearsJob Type: ContractJob SummaryWe are seeking an experienced Senior Data Engineer with strong hands-on expertise in PySpark, Databricks, Python, and SQL to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have extensive experience building data pipelines, processing large datasets, developing ETL/ELT workflows, and optimizing data processing performance.The candidate must possess strong programming and problem-solving skills, with the ability to write clean, efficient, and production-ready code.Key ResponsibilitiesDesign, develop, and maintain scalable data pipelines using PySpark and Databricks.Write, test, debug, and optimize Python, PySpark, and SQL code.Build and maintain ETL/ELT processes for structured and unstructured data.Develop data transformations and workflows to process large volumes of data.Work with Databricks notebooks, Delta Lake, and Spark-based data processing frameworks.Optimize Spark jobs, troubleshoot performance issues, and improve pipeline efficiency.Implement data quality checks, validation, error handling, and monitoring.Integrate data from multiple sources, including databases, APIs, cloud storage, and enterprise systems.Collaborate with data architects, analysts, and business stakeholders to deliver reliable data solutions.Follow coding standards, version control practices, testing procedures, and CI/CD processes.Required Skills10+ years of experience in data engineering or related software/data development roles.Strong hands-on experience with Python and PySparkPractical experience developing data solutions using DatabricksStrong SQL skills, including complex queries, joins, aggregations, and query optimization.Experience building and maintaining ETL/ELT data pipelines.Knowledge of Apache Spark architecture, distributed processing, and performance tuning.Experience with Delta Lake and lakehouse architecture.Strong coding, debugging, troubleshooting, and problem-solving skills.Experience with Git and software development best practices.Familiarity with cloud platforms such as AWS, Azure, or GCP.Strong communication and collaboration skills.Preferred QualificationsExperience with Azure Databricks, AWS Databricks, or another cloud-based data platform.Knowledge of data warehousing, dimensional modeling, and data lake architecture.Experience with orchestration tools such as Airflow, Azure Data Factory, or Databricks Workflows.Familiarity with CI/CD, automated testing, and deployment practices.Experience working in Agile development environments.