Data Engineer

We are seeking a Senior Data Engineer to design, develop, and maintain scalable data pipelines and data ingestion processes using modern Big Data technologies. The ideal candidate will have strong experience in data aggregation, transformation, validation, quality management, ETL, and troubleshooting across large-scale data environments.

Roles and Responsibilities

• Design, develop, and maintain complex data pipelines and data ingestion processes for data lake environments.

• Develop data transformation processes to aggregate, standardize, link, validate, and load data from multiple sources.

• Build and maintain data engineering solutions using Spark, Scala, Hadoop, Hive, SQL/T-SQL, and Shell scripting.

• Perform data validation, quality checks, analysis, and troubleshooting to ensure data accuracy and integrity.

• Analyze data issues, perform root-cause analysis, and implement solutions to improve data quality and pipeline performance.

• Develop tools and processes to improve data processing efficiency, reliability, and performance.

• Work with technical and operations teams to troubleshoot complex issues involving databases, Linux environments, storage, and servers.

• Support production environments and resolve critical data and pipeline issues as needed.

• Generate statistical and operational reports and perform program/data validation.

• Mentor junior data engineers on data engineering, troubleshooting, and data quality best practices.

• Collaborate with cross-functional teams in an Agile/SAFe environment to deliver projects within defined timelines.

Required Qualifications

• 8–10 years of overall industry experience with a Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related field.

• 5+ years of experience in data aggregation, standardization, data linking, quality checks, and reporting.

• 5+ years of experience with Big Data technologies such as Hadoop and Spark.

• 5+ years of experience with relational databases such as Oracle and Microsoft SQL Server.

• Strong experience with SQL, advanced SQL queries, ETL processes, and data integration tools.

• Strong understanding of Linux environments, shell scripting, and file systems.

• Hands-on experience with Spark and Scala.

• Strong data analysis, debugging, troubleshooting, and root-cause analysis skills.

• Experience working in an Agile development environment.

Preferred Qualifications

• Experience working with healthcare data or in a healthcare data operations environment.

• Experience with Hive and additional Big Data technologies.

• Experience with data lake and large-scale data ingestion platforms.

• Experience working in Scaled Agile Framework (SAFe) environments.

• Availability to support production issues outside standard business hours when required.

• EST time-zone availability is preferred.