Senior Data Engineer

Design, develop, and maintain scalable data processing pipelines and workflows using frameworks such as Apache Spark, PySpark, and Apache Beam. Build and maintain microservices in Python that serve data-driven features in production. Develop internal tools to support CI/CD pipelines, experiment tracking, and data versioning. Collect, process, and integrate large datasets from multiple sources, including databases, file systems, and APIs. Ensure data integrity, consistency, and quality through robust validation and monitoring processes. Optimize data systems for performance, scalability, and high availability. Implement best practices for data security, access control, and privacy. Collaborate with data scientists, analysts, and engineers to support analytics and ML workflows. Lead complex migration initiatives involving the transition from on-premise Cloudera environments to cloud-native platforms, ensuring zero data loss and minimal downtime. Requirements Strong understanding of distributed systems and modern data architectures. Proven, hands‑on experience with large-scale data platform migrations, specifically transitioning from Cloudera (CDH/HDP) to either Snowflake or Databricks. Deep technical expertise in building and orchestrating a high‑performance data platform from scratch. 5+ years of professional experience in software engineering or data engineering. Strong software engineering skills with Python in large‑scale, high‑performance production environments. Hands‑on experience with Spark/PySpark and other big data frameworks. Core Competencies Demonstrates expertise in designing and maintaining scalable data processing pipelines using frameworks like Apache Spark and PySpark, with a strong focus on data integrity and performance optimization. Proven ability to lead data platform migrations and collaborate effectively with cross‑functional teams to support analytics and machine learning workflows. Highest-signal resume keywords Apache Spark PySpark Data Platform Migration Python Software Engineering Distributed Systems ATS Optimization Keywords Hard Skills Data Processing Pipelines Microservices Development CI/CD Pipelines Data Integration Data Validation Performance Optimization Data Security Access Control Cloud‑Native Platforms Big Data Frameworks Soft Skills Collaboration Problem‑Solving Communication Industry Keywords Data Architecture Data Quality Data Versioning Analytics Workflows Machine Learning Tools & Technologies Apache Beam Cloudera Snowflake Databricks #J-18808-Ljbffr Jobtailor