Sr. Data Engineer

Role Description

Our Client is looking for an experienced, motivated, and self-driven candidate who can contribute towards various aspects of taking the Live Analytics offering to the next level and build new data products for our clients. This involves design, development, and production support of the systems as well as working with product teams and engineering teams in shaping the future of the application.

• Write new ETLs to support both real-time and new features data and reporting products.
• Work with product teams to gather and understand requirements, discuss approaches, design solutions based on the requirements, and create design documents for implementation keeping in mind costs, capacity, shelf-life, and choosing the right tools and technologies.
• Design and develop peripheral systems and processes - monitoring and alerting services, ETL processes, automated offline workflows, automated build and deployment pipelines, systems for gathering application performance metrics, ensuring efficiency in cost.
• Ensure best practices in system development ranging from test coverage, use of enterprise standards, adherence to target architecture, high-availability and scalability, low latency, and fault-tolerance, thus ensuring delivery of high-quality systems.
• Create detailed documentation on systems as well as processes ranging from comprehensive overview of systems, runbooks for developing, monitoring and troubleshooting system issues and recommendations on future improvements.
• Managing project delivery from a day-to-day standpoint - participating in daily agile stand-ups, escalating blockers to stakeholders, ensuring timely completion of tasks & overall project & sprint goals. Communicating updates to stakeholders and working with them on timelines.

Qualifications

• Bachelor's degree in Computer Science or related technical field. (NOT REQUIRED TO HAVE A DEGREE)
• Experience building Data Lakes, Data Warehouse using big data and cloud stack.
• 5+ years of experience on software development using Agile, dev ops that include designing, developing, and implementing large-scale data engineering solutions.
• Expert Level Proficiency in Spark streaming and spark concepts.
• Expert level proficiency in Python Programming language and PySpark with at least 4 years of experience.
• Experience working with Kafka streams and creating ingestion ETLs.
• Expert level proficiency in ANSI SQL / Spark SQL with at least 5 years of experience.
• Experience working with diverse data sources and data formats (json, parquet, csv, psv, delta).
• Experience working with cloud services such as AWS along with cloud-based data sources (S3).
• Experience building ETL with batch and real-time stream processing.
• Experience using version control system such as git and CI/CD.

Requirements

• Experience with building workflows and ETLs in the databricks.
• Experience ingesting data produced by real-time systems like Kafka.
• Guiding and mentoring skills - guiding and mentoring junior members - helping them understand technologies, the project, best practices and in general setting them up for success.
• Experience building and deploying ML models using ML Flow.

Benefits

• Comprehensive medical benefits
• Competitive pay
• 401(k) retirement plan
• ...and much more!