Big Data Engineer

Big Data Engineer

Immediate need for a talented Big Data Engineer. This is a Fulltime opportunity with long-term potential and is located in Detroit MI (Hybrid). Job ID:26-13596 Pay Range: $105000 - $110000/annum+ 5% org bonus + 5% performance bonus. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).

Key Responsibilities:

• Design, develop, and maintain scalable big data pipelines and ETL workflows using AWS Glue and Python.
• Build and optimize Spark-based Glue jobs for efficient data processing and transformation.
• Develop and manage real-time data streaming solutions using Apache Kafka.
• Create and support event-driven serverless applications using AWS Lambda.
• Design, optimize, and maintain Client data warehouse solutions, including schema design, query optimization, and performance tuning.
• Write advanced SQL queries for large-scale data transformation, reporting, and analytics.
• Manage and integrate AWS cloud services such as S3, CloudWatch, SES, and IAM.
• Implement CI/CD pipelines and infrastructure automation using Terraform and Git Actions.
• Monitor data workflows and troubleshoot issues related to data quality, system performance, and scalability.
• Collaborate with cross-functional teams including developers, analysts, and business stakeholders to deliver data-driven solutions.
• Ensure data security, reliability, and operational efficiency across cloud-based data platforms.
• Participate in code reviews, technical discussions, and continuous improvement initiatives.
• Work effectively in a fast-paced hybrid environment with minimal supervision.

Key Requirements and Technology Experience:

• Strong Python experience (hands-on coding required)
• AWS Glue experience with Spark-based Glue jobs
• Apache Kafka experience for real-time streaming
• Experience with Client data warehousing
• Advanced SQL skills
• Experience with AWS Lambda
• Experience with Terraform
• Experience with Git Actions / CI-CD
• AWS Glue: Proficiency in developing and managing Spark-based Glue jobs.
• Client: Strong experience in Client data warehousing, including schema design, query optimization, and performance tuning.
• Kafka: Hands-on experience with Kafka for real-time data streaming and integration.
• AWS Lambda: Experience in building serverless functions for event-driven workflows. Programming: Proficiency in Python - SQL: Advanced SQL skills for data transformation and analysis.
• Cloud Infrastructure: Familiarity with AWS services such as S3, CloudWatch, SES and IAM.
• CI/CD: Familiarity with Terraform and Git actions.
• Strong problem-solving and analytical skills.
• Excellent communication and collaboration abilities.
• Ability to work in a fast-paced, dynamic environment with minimal supervision.
• Spark optimization experience
• Real-time data pipeline architecture
• Event-driven/serverless architecture
• AWS certification
• Experience in enterprise-scale data engineering
• Strong communication/client-facing experience
• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field is preferred