Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Sr. Data Engineer

Temporary

Long Finch Technologies

Job Description: Sr. Data Engineer – Data Ingestion Framework

Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as

the data Lakehouse storage layer and Snowflake as the target data platform.

 

Experience: 10–12 Years
Location: Remote – USA (Onshore)
Work Arrangement: Remote or Hybrid (preferred local to Boston)
Engagement: Data Platform Modernization / Data Ingestion Framework

Position Overview

We are looking for an experienced Lead Data Engineer with strong expertise in Python and DBT to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing ETL capabilities with an open-source technology stack leveraging Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.

Key Responsibilities

1. Data Ingestion Framework Development

  • Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.
  • Design modular and extensible components to support multiple source systems.
  • Develop metadata-driven and configuration-based ingestion capabilities.
  • Build reusable connectors and adapters for different source technologies.
  • Implement full and incremental data ingestion patterns, including CDC where applicable.
  • Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.

2. Data Integration & Lakehouse Architecture

  • Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.
  • Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.
  • Design and implement data lakehouse ingestion patterns using AWS S3.
  • Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.
  • Implement schema handling, data validation, partitioning, and data movement strategies.

3. Python & dbt Engineering

  • Develop production-grade Python applications and reusable ingestion libraries.
  • Build and maintain dbt models for data transformation and integration.
  • Implement dbt testing, documentation, incremental models, and reusable macros.
  • Optimize Python-based ingestion processes and dbt transformations for performance and scalability.
  • Follow engineering best practices for modularity, maintainability, and code quality.

4. Workflow Orchestration

  • Implement and manage workflow orchestration using Control-M and Dagester.
  • Develop reusable workflows for scheduling, dependency management, retries, and pipeline monitoring.
  • Integrate orchestration workflows with Python ingestion components and Snowflake.

5. Testing, Deployment & Production Support

  • Develop unit tests and integration tests for ingestion framework components.
  • Support migration and deployment across DEV, SIT, UAT, and PROD environments.
  • Collaborate with QA teams to perform functional, integration, regression, and performance testing.
  • Troubleshoot ingestion failures, data discrepancies, and performance bottlenecks.
  • Support production deployment, stabilization, documentation, and knowledge transfer.

6. Technical Leadership & Collaboration

  • Work closely with the onshore solution architect, customer stakeholders, and offshore engineering teams.
  • Provide technical guidance, code reviews, and engineering best practices.
  • Participate in architecture discussions, technical design reviews, and sprint planning.
  • Mentor developers and ensure adherence to established engineering standards.

Required Technical Skills

  • Python: Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.
  • DBT: Strong hands-on expertise in DBT Core, including models, macros, tests, incremental processing, and deployment practices.
  • Data Engineering: Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.
  • Snowflake: Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.
  • AWS: Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.
  • Data Integration: Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.
  • SQL: Advanced SQL development and query optimization skills.
  • CI/CD: Experience with Git-based version control, automated builds, testing, and deployment pipelines.

Good-to-Have Skills

  • Hands-on experience with  Dagster for data orchestration and asset-based pipeline development.
  • Experience with utilizing Control-M JOB scheduler.
  • Experience with Apache Iceberg and open table formats.
  • Experience implementing metadata-driven ingestion frameworks.
  • Experience migrating ETL workloads from SnapLogic or similar integration platforms.
  • Experience with Docker, Terraform, and infrastructure automation.
  • Experience implementing data quality, observability, and monitoring solutions.
  • Experience working in an Agile environment utilizing SCRUM

Qualifications

  • 10–12 years of overall experience in data engineering and data integration.
  • Strong architectural and hands-on engineering capabilities.
  • Experience working on enterprise data modernization initiatives.
  • Excellent communication and stakeholder management skills.
  • Ability to work independently in a distributed onshore/offshore delivery model.

Mandatory Skills

  1. Expert-level Python development.
  2. Strong hands-on dbt experience.
  3. Enterprise data ingestion framework development.
  4. Snowflake and AWS S3 experience.
  5. Strong SQL and data integration experience.

Preferred:   Dagster, Control-M, Apache Iceberg, and SnapLogic migration experience.

 

 

Vacancy posted 1 hour ago
Similar jobs that could be interesting for youBased on the Sr. Data Engineer in Boston, MA vacancy
  •  ...exceptional team that embodies a true startup mindset. The Analytics Engineering Team at ABCorp partners closely with Product Management, Engineering, Sales, and Customer Success to build the data foundations that drive product strategy and business decisions.... 
    Senior
    Worldwide

    ABCorp NA Inc.

    West Roxbury, MA
    24 days ago
  •  ...NessContact: Meghana GorusuCompany: SRI Tech SolutionsJob Title: Sr Data EngineerLocation: Boston, MA (3-4 days a week onsite)Skill Set:...  ...SQL and PythonJob Description:We are looking for a Senior Data Engineer with added emphasis on data analysis and Salesforce knowledge,... 
    Senior
    Work at office
    3 days per week

    SRI Tech

    Boston, MA
    4 days ago
  • $116.2k - $229.1k

    Position Summary Join Deloitte’s AI & Engineering practice and help organizations transform enterprise technology platforms, modernize data environments, and unlock value through innovation. As a Databricks Engineer in our AI & Data practice, you will design, build... 
    Senior
    Local area
    Visa sponsorship

    Deloitte

    Boston, MA
    3 days ago
  • $150k - $165k

     ...your opportunity to grow professionally in a dynamic, global environment, surrounded by forward-thinking peers.The role of Senior Data Engineer is essential for the successful modernization of our data platforms and practices. The Senior Data Engineer will collaborate... 
    Senior
    Full time

    Goodwin Procter

    Boston, MA
    3 days ago
  • $63 - $67 per hour

     ...Location: Boston, MA Salary: $63.00 USD Hourly - $67.00 USD Hourly Description: Our client is currently seeking a Senior Data Engineer Position Summary We're seeking a Senior Data Engineer with a strong backend foundation to build and operate data pipelines... 
    Senior
    Hourly pay

    Judge Group, Inc.

    Boston, MA
    1 day ago
  •  ...Job Description Job Description Senior Analytics Engineer – Data Architecture, dbt, SQL & AI Job Title: Senior Analytics Engineer Job ID: 1006 Location: Boston, MA Duration: 12 Months Contract Work Arrangement: Remote/Hybrid – Preference for candidates... 
    Senior
    Contract work
    Local area
    Remote work

    Sigma Inc

    Boston, MA
    20 days ago
  • $120k - $187k

     ...: Information TechnologyLocation: Boston, MA, United StatesWorkplace: HybridSchedule: Full timePosted: 2026-10-27As a Senior BI Data Engineer, you will design, build, and optimize enterprise-scale data solutions that power analytics and business decision-making across Boston... 
    Senior
    Full time
    Work at office

    The Boston Beer Company

    Boston, MA
    3 days ago
  •  ...Vice President, Senior Data Engineer About the Company Mission-driven provider of tax credit syndication services Industry Real Estate Type Privately Held Founded 2001 Employees 51-200 Specialties lihtc tax credit syndication low income... 
    Senior

    Confidential

    Boston, MA
    2 days ago
  • $140k - $160k

     ...company that enables organizations to share and analyze sensitive data at speed without compromising security or control. VIA’s...  ...impressive mission requires an equally impressive Senior Data Analytics Engineer. As a Senior Data Analytics Engineer at VIA, you'll play a... 
    Senior
    Summer work
    Remote work
    Work from home
    Flexible hours

    VIA

    Somerville, MA
    2 days ago
  • $98.5k - $141.5k

     ...institutions who rely on us to help them build more secure and prosperous futures. THE ROLEThe Investment Data Management Office is seeking a Senior Investment Data Engineer with strong SQL and data quality assurance experience to support the implementation, testing, and... 
    Senior
    Full time
    Work at office
    Local area
    Remote work
    Flexible hours

    MFS Investment Management

    Boston, MA
    4 days ago
  • $150k - $180k

     ...humanitarian missions worldwide. With 1,700+ team members, 1,500+ AI/data experts, and 100+ prime contracts, we deliver at scale and with...  ...Summary Credence has an immediate need for a Senior AI Data Engineer to join our growing AI and Automation practice. You will be a... 
    Senior
    Temporary work
    Immediate start
    Worldwide

    Credence

    Boston, MA
    9 days ago
  •  ...passionate about getting physicians and healthcare providers away from the keyboard and back to clinical care. Overview The Senior Data Engineer is a member of the Data Platform team, reporting to the Director of Machine Learning Engineering and Data. The Data Platform... 
    Senior
    Full time
    Temporary work
    Work at office

    CodaMetrix

    Boston, MA
    3 days ago
  • Job Title: Sr.Data Engineer and Data Engineer Location: CST or EST PS Offfice (Boston, Atlanta, Houston, Chicago, New York) must be 40miles to an office and able to pick up laptop/id verification day 1-3 days a week onsite Length: 6-10 months    Build and maintain... 
    Senior
    Contract work
    Work at office
    3 days per week
    1 day per week

    Scadea

    Boston, MA
    17 days ago
  • $132.9k - $182.7k

     ...our best and know that we’re making a difference. As a Senior Structural Engineer professional based within our Advanced Manufacturing team, you’ll work on a variety of projects related to Data Centers, including new builds, retrofits, colocations, etc. You will be performing... 
    Senior
    Full time
    Remote work

    Jacobs

    Boston, MA
    4 days ago
  • $171.2k - $228.2k

     ...in their home. We're investing in a multi-quarter Enterprise Data Platform program to move our data and analytics stack onto a...  ...modern lakehouse. We're looking for a Senior Manager, Data Platform Engineering to lead the team that builds, operates, and evolves that... 
    Senior
    Work at office

    SimpliSafe

    Boston, MA
    28 days ago
  • $100k - $150k

     ...Job Description As a Senior Data Scientist / Product Engineer at Kearney Activate, you will play a key role in delivering secure, cloud-hosted data and analytics solutions for clients within Kearney’s Mobility, Defense and Advanced Industrials practice. This role is designed... 
    Senior
    Full time
    H1b
    Immediate start
    Home office
    Visa sponsorship
    Flexible hours

    Kearney

    Boston, MA
    11 days ago
  • $179k - $236k

     ...intervention points for new therapies. Vesalius was founded in 2019 by Flagship Pioneering. Role Overview We are seeking a Principal Data Engineer to lead the architecture and evolution of the data platform underpinning our human genetics, multi-omics, and high-content... 
    H1b
    Relocation
    Work visa

    Vesalius Therapeutics

    Cambridge, MA
    3 days ago
  • $152k - $228.5k

     ...Principal Data EngineerWith 1,000+ intelligence professionals serving over 1,900 clients worldwide, Recorded Future is the world's most...  ...Protection data streams. We are looking for a Principal Data Engineer to drive improvements in the Digital Risk Protection department.... 
    Full time
    Worldwide

    Venturefizz Product Management Community

    Somerville, MA
    14 hours ago
  • Senior to Principal Data Engineer (SQLserver, Financial Services), Boston, MA. Hybrid, min 3 days week in office, 135k to 175k base on experienceCompensation Commensurate with experience, EOE, diverse highly educated culture Candidates must be a U.S. citizen or national... 
    Senior
    Permanent employment
    Contract work
    Work at office
    Local area
    3 days per week

    BlueSkyClarity

    Boston, MA
    4 days ago
  •  ...Control Tower team at State Street. The role focuses on delivering reporting and analytics by designing Power BI dashboards, building data models, and performing ETL and data transformations to turn data into actionable insights. Key Responsibilities Design, develop, and... 
    Senior
    Temporary work

    State Street Corp

    Quincy, MA
    4 days ago
  • $119k - $299.93k

     ...Senior Manager Job Description & Summary The Opportunity As part of the AI Engineering team within the Digital Assurance & Technology team, you will play a pivotal role in transforming raw data into actionable insights, enabling informed decision-making and driving business... 
    Senior
    Full time
    H1b

    PwC

    Boston, MA
    5 days ago
  • $115k - $150k

    About the jobSenior Associate, Data Products | Finance | Boston, MAOverviewJoin a high-performing investment-focused organization as...  ...platforms.Required Experience6+ years of experience in Data Engineering, Data Platform Engineering, or Software Engineering.Deep expertise... 
    Senior
    Work at office
    3 days per week

    Huxley Associates

    Boston, MA
    2 days ago
  • $300k

    Senior Data EngineerOur client is one of the world's largest privately held investment management firms, overseeing hundreds of billions...  ...and best practices across the organizationPartner closely with engineering, data, business, and investment stakeholders to translate... 
    Senior

    Huxley Associates

    Boston, MA
    4 days ago
  • Description At Liberty Mutual Investments Technology, we are seeking a skilled Senior Data Engineer who is ready to take on exciting challenges, contribute to our cloud-native data platform, and make a real impact across our investment insights. You will build modern... 
    Senior
    Work at office
    Local area

    Liberty Mutual

    Boston, MA
    1 day ago
  • $140k - $200k

    Boston, MAHybridFull Time$140k - $200k Job Description A rapidly growing scientific data and AI technology company is seeking a Senior Scientific Data Engineer to join its fully remote team. This role offers the opportunity to work at the intersection of data engineering... 
    Senior
    Full time
    Temporary work
    Immediate start
    Remote work

    Motion Recruitment

    Boston, MA
    4 days ago
  • $175.5k - $180k

     ...technical owner, designing and leading the implementation of scalable data architectures for cutting-edge AI and agentic systems. You will...  .... You will also spearhead the design of complex feature engineering workstreams, ensuring our data assets are not only robust but also... 
    Senior
    Apprenticeship
    Easy work

    McKinsey & Company

    Boston, MA
    4 days ago
  • $195.3k - $199.2k

     .... In this role you will design, develop, and maintain scalable data pipelines and systems using Databricks on Azure (among other environments...  ...sharing capabilities. You will be leading and executing data engineering projects from inception to completion, ensuring timely delivery... 
    Senior
    Apprenticeship
    Easy work

    McKinsey & Company

    Boston, MA
    2 days ago
  • Senior Data Engineer - Boston, United StatesWe are partnering with our client, a leading institutional asset manager based in Boston's financial district, to appoint a Senior Data Engineer to join our client's team. In this permanent role, you will be instrumental in building... 
    Senior
    Permanent employment
    Work at office
    Remote work
    Monday to Friday

    Huxley Associates

    Boston, MA
    4 days ago
  • Senior Investment Data Engineer, Boston, MA, Hybrid, 2 days in, 150k to 190kCompensation Commensurate with experience, bonus, benefits, EOE, diverse highly educated cultureCandidates must be a U.S. citizen or national, refugee, asylum, or lawful permanent resident. Principals... 
    Senior
    Permanent employment
    Work experience placement
    Casual work
    Work at office

    BlueSkyClarity

    Boston, MA
    4 days ago
  • As a Senior Data Engineer, CASM Platform, you will: Data Integration, API Development: Integrate diverse cybersecurity data sources using variety of API mechanisms and to standardize and streamline data across the data and user planes.Build and maintain data APIs for seamless... 
    Senior
    Contract work

    Cedent Consulting

    Boston, MA
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Sr. Data Engineer. Be the first to apply!