Data Engineer

Data Engineer

We are seeking a Data Engineer for a role in the financial services industry. The position involves authoring, debugging, and optimizing complex data scripts using Python and advanced SQL. The candidate will work with Apache Airflow to design, schedule, and monitor data pipelines and will be responsible for improving existing data pipelines by resolving failures and optimizing resource utilization.

Key Responsibilities

• Author, debug, and optimize complex data scripts using Python and advanced SQL queries.
• Utilize Apache Airflow for designing, scheduling, and monitoring complex data pipelines and DAGs.
• Manage SQL-based database systems, including schema design, query performance tuning, and data warehousing.
• Implement ETL/ELT data integration techniques, data transformation, and manage distributed storage systems.
• Install, configure, and maintain data pipeline utilities, custom Airflow providers, and database connectors.
• Create and maintain technical documentation for data architecture, DAG workflows, and operational procedures.
• Improve existing data pipelines by resolving recurring failures, eliminating bottlenecks, and optimizing resource utilization.
• Identify opportunities to innovate, automate, and scale data delivery across the organization.
• Perform Root Cause Analysis and Incident Management for data quality issues, DAG failures, and database outages.

Required Qualifications

Experience: 4-7 years of data engineering experience is required.

Technical Skills

• Experience with cloud-based data platform infrastructure and containerization technologies such as Docker or Kubernetes.
• Understanding of modern data stack technologies and ability to assist with cloud-based data lake or warehouse integration efforts.
• Proficiency in Agile development methodologies and CI/CD deployment pipelines for code and workflows.
• Ability to thrive in a sense-of-urgency environment and leverage data engineering best practices.