Senior Data Engineer

Job title: Senior Data Engineer

Location: Irving Place, New York – Hybrid (Weekly 3 days onsite and 2 days remote)

Rate: $65/hour on W2

Type: Contract

Must: In-person interview and 10+ Years of experience

Key Responsibilities

Data Platform Engineering

• Design, build, and maintain scalable batch and real-time data pipelines.
• Develop and optimize data ingestion, transformation, and processing frameworks for structured, semi-structured, and unstructured datasets.
• Implement modern Lakehouse architectures utilizing Databricks, Delta Lake, and Medallion (Bronze, Silver, Gold) design patterns.
• Build data solutions that support enterprise analytics, reporting, and machine learning initiatives.
• Ensure data quality, governance, lineage, security, and compliance across data ecosystems.

Big Data & Streaming Solutions

• Develop distributed data processing applications using PySpark and Spark SQL.
• Build and maintain streaming pipelines using Kafka, Spark Structured Streaming, and AWS Kinesis.
• Design fault-tolerant, scalable systems capable of processing large data volumes with low latency.
• Optimize workload performance through partitioning strategies, clustering, caching, and query tuning.

Cloud & Lakehouse Architecture

• Architect and implement cloud-based data solutions on AWS.
• Utilize AWS services including S3, EMR, EC2, Athena, Redshift, RDS, Lambda, IAM, SNS, and SQS.
• Design data storage and processing strategies that maximize reliability while minimizing operational costs.
• Support migration initiatives from traditional Hadoop and EMR environments to modern cloud-native platforms.

Required Qualifications

• Bachelor's or Master's degree in Computer Engineering, Computer Science, Information Systems, or a related field.
• 10+ years of experience in software engineering, data engineering, or big data platform development.
• Strong experience designing and implementing enterprise-scale data pipelines.
• Hands-on expertise with:
• Python
• PySpark
• Spark SQL
• SQL
• Kafka
• Databricks
• Delta Lake
• Snowflake
• Hive
• Experience building data solutions on AWS cloud platforms.
• Strong understanding of distributed computing, data modeling, and large-scale data processing.
• Experience with Git-based development workflows and CI/CD practices.
• Excellent analytical, troubleshooting, and problem-solving skills.