Data Engineer

Data Engineer – Pipeline Operations & Incident ResponseThis role is heavily focused on maintaining and stabilizing large-scale data pipelines in a production environment. The majority of time is spent troubleshooting and resolving issues across existing data workflows rather than building new systems. Early success in this position looks like gaining enough familiarity with the platform, data flows, and key stakeholders to independently diagnose and resolve pipeline failures across multiple environments.Key Responsibilities:Investigate and resolve data pipeline failures across multiple production environmentsPerform root cause analysis on data quality and pipeline performance issuesApply targeted code fixes and adjustments to restore pipeline functionalityMonitor pipeline health and respond to alerts within defined SLAsSupport and maintain existing ETL processes rather than developing new onesRefactor pipelines to resolve performance issues such as memory constraints or inefficient processingCoordinate with upstream data providers and internal teams to resolve data ingestion issuesEscalate issues when access, ownership, or dependencies fall outside immediate controlDay-to-Day Breakdown:~85–90%: Debugging, incident response, and pipeline issue resolution~5–10%: Monitoring, validation, and health checks~5–10%: Minor code updates, optimizations, and pipeline adjustmentsWork is centered on fixing and stabilizing existing pipelines, not building new ones from scratch.Technical Environment:Predominantly batch-based ETL pipelines (incremental processing is common)High-volume pipeline ecosystem spanning multiple data domains and environmentsMix of code-driven pipelines and low-code/visual pipeline toolsStreaming pipelines are minimalRequired Technical Skills:Strong experience with large-scale data engineering and ETL/ELT workflowsProficiency in Python and distributed data processing frameworks (PySpark preferred)Solid understanding of dataframes and data manipulation at scaleExperience troubleshooting production data pipelines and debugging failuresKnowledge of relational databases and SQL fundamentalsFamiliarity with distributed computing conceptsAdditional Technical Exposure:Experience with Java or similar languages (C++ acceptable alternative)Ability to diagnose and resolve memory/performance issues in distributed jobsExposure to visual pipeline tools or data workflow platforms is helpfulBasic understanding of networking concepts and API-based data ingestionOperational Environment:Engineers support a large number of pipelines across multiple environments simultaneouslyWork is highly reactive, driven by incoming alerts and data incidentsEngineers are expected to quickly assess and troubleshoot pipelines they have not previously worked onHigh alert volume, with multiple issues often tied to common root causesCollaboration:Frequent interaction with data providers to resolve source data issuesRegular coordination with cross-functional technical teams on pipeline failuresOccasional engagement with end users reporting data discrepanciesOn-Call & Incident Response:Rotating on-call schedule supporting different pipeline groupsSome rotations may include off-hours alerts tied to overnight pipeline processingMajority of incidents handled during business hours, with occasional escalation scenariosEngineers are expected to own resolution when possible and coordinate when dependencies existIdeal Candidate Background:Strong foundation in data engineering within production environmentsExperience supporting operational data systems rather than purely building new solutionsComfortable working in high-volume, incident-driven environmentsAble to quickly understand and troubleshoot unfamiliar systemsHands-on experience with distributed data processing and large datasets