Sr. Data Engineer with Python and Azure

Job Description

Sr. Data Engineer

\n

Location: Chicago

\n

Hybrid role- candidates must be willing to go onsite three days a week

\n

Rate: $70-$80/hour

\n

H1b’s are permitted but MUST be willing to sign a 90 day release.

\n

\n

The Planet Group has partnered with a Chicago area firm to locate a Sr. Data Engineer to build and support scalable data pipelines that power business intelligence, analytics, and downstream applications.

\n

\n

This hands-on role is ideal for someone with strong Python, Apache Spark, and SQL skills who enjoys solving complex data challenges and delivering reliable production solutions.

\n

You will collaborate with technical and business partners to translate data requirements into efficient workflows, maintain data quality, and improve pipeline performance.

\n

\n

Key Responsibilities

\n

\n
• Design, build, and maintain production ETL pipelines that extract, transform, and integrate large datasets.
\n
• Develop data processing workflows using Python and Apache Spark.
\n
• Write and optimize complex SQL queries to support data transformation, validation, and analysis.
\n
• Partner with cross-functional teams to understand source data, define requirements, and deliver usable datasets.
\n
• Implement data quality checks to improve accuracy and consistency across pipelines.
\n
• Investigate pipeline failures, troubleshoot processing issues, and resolve performance bottlenecks.
\n
• Improve the scalability, reliability, and maintainability of existing data solutions.
\n
• Follow engineering best practices for version control, code documentation, and collaborative development.
\n
• Participate in an agile team environment and communicate progress, issues, and technical decisions clearly.
\n

\n

Required Qualifications

\n

\n
• 5+ years of hands-on experience developing and maintaining ETL pipelines in production environments.
\n
• Strong Python programming skills for data processing, transformation, and automation.
\n
• Practical experience using Apache Spark to process large datasets.
\n
• Advanced SQL skills, including complex queries and data transformations.
\n
• Solid understanding of data engineering principles, including scalability, reliability, data quality, and performance optimization.
\n
• Experience troubleshooting production data workflows and resolving technical issues.
\n
• Strong communication, problem-solving, and collaboration skills.
\n

\n

Preferred Qualifications

\n

\n
• Experience with Microsoft Azure data services, particularly Azure Data Factory.
\n
• Experience developing or running Spark workloads in Databricks.
\n
• Familiarity with Power BI and how data pipelines support reporting and analytics.
\n
• Exposure to infrastructure as code tools, such as Pulumi.
\n

\n