Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Sr. Data Engineer

Longfinch Technologies

Job Description:

Sr. Data Engineer – Data Ingestion Framework

Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as

the data Lakehouse storage layer and Snowflake as the target data platform.

Engagement:

Data Platform Modernization / Data Ingestion Framework

Position Overview

We are looking for an experienced Lead Data Engineer with strong expertise in Python and DBT to design and develop a scalable, reusable, and technology-agnostic data ingestion framework.

The role involves replacing existing ETL capabilities with an open-source technology stack leveraging Python, DBT, and orchestration technologies such as Dagster or Control-M. The solution will utilize AWS S3 as the data Lakehouse storage layer and Snowflake as the target data platform.

The ideal candidate should have hands-on experience building enterprise-grade data ingestion frameworks, developing reusable connectors, implementing metadata-driven pipelines, and integrating heterogeneous source systems with modern cloud data platforms.

Key Responsibilities

  1. Data Ingestion Framework Development

Develop a reusable, scalable, and technology-agnostic data ingestion framework using Python.

Design modular and extensible components to support multiple source systems.

Develop metadata-driven and configuration-based ingestion capabilities.

Build reusable connectors and adapters for different source technologies.

Implement full and incremental data ingestion patterns, including CDC where applicable.

Implement robust error handling, logging, auditing, retry, and reprocessing mechanisms.

  1. Data Integration & Lakehouse Architecture

Develop ingestion pipelines to integrate data from heterogeneous source systems into AWS S3 and Snowflake.

Implement data ingestion from SQL Server, Oracle, PostgreSQL, REST APIs, SFTP, AWS S3, and other source systems.

Design and implement data lakehouse ingestion patterns using AWS S3.

Work with file formats such as CSV, JSON, Parquet, and other structured/semi-structured formats.

Implement schema handling, data validation, partitioning, and data movement strategies.

  1. Python & dbt Engineering

Develop production-grade Python applications and reusable ingestion libraries.

Build and maintain dbt models for data transformation and integration.

Implement dbt testing, documentation, incremental models, and reusable macros.

Optimize Python-based ingestion processes and dbt transformations for performance and scalability.

Follow engineering best practices for modularity, maintainability, and code quality.

  1. Workflow Orchestration

Implement and manage workflow orchestration using Control-M and Dagester.

Develop reusable workflows for scheduling, dependency management, retries, and pipeline monitoring.

Integrate orchestration workflows with Python ingestion components and Snowflake.

  1. Testing, Deployment & Production Support

Develop unit tests and integration tests for ingestion framework components.

Support migration and deployment across DEV, SIT, UAT, and PROD environments.

Collaborate with QA teams to perform functional, integration, regression, and performance testing.

Troubleshoot ingestion failures, data discrepancies, and performance bottlenecks.

Support production deployment, stabilization, documentation, and knowledge transfer.

  1. Technical Leadership & Collaboration

Work closely with the onshore solution architect, customer stakeholders, and offshore engineering teams.

Provide technical guidance, code reviews, and engineering best practices.

Participate in architecture discussions, technical design reviews, and sprint planning.

Mentor developers and ensure adherence to established engineering standards.

Required Technical Skills

Python:

Expert-level hands-on experience in Python development, including object-oriented programming, reusable libraries, APIs, error handling, and performance optimization.

DBT:

Strong hands-on expertise in DBT Core, including models, macros, tests, incremental processing, and deployment practices.

Data Engineering:

Extensive experience building enterprise-scale ETL/ELT pipelines and reusable data ingestion frameworks.

Snowflake:

Strong experience with Snowflake architecture, data loading, stages, Snowpipe, SQL, and performance optimization.

AWS:

Hands-on experience with AWS S3, IAM, and cloud-based data integration patterns.

Data Integration:

Experience integrating relational databases, REST APIs, SFTP, and cloud-based storage systems.

SQL:

Advanced SQL development and query optimization skills.

CI/CD:

Experience with Git-based version control, automated builds, testing, and deployment pipelines.

Good-to-Have Skills

Hands-on experience with Dagster

for data orchestration and asset-based pipeline development.

Experience with utilizing Control-M JOB scheduler.

Experience with Apache Iceberg and open table formats.

Experience implementing metadata-driven ingestion frameworks.

Experience migrating ETL workloads from SnapLogic or similar integration platforms.

Experience with Docker, Terraform, and infrastructure automation.

Experience implementing data quality, observability, and monitoring solutions.

Experience working in an Agile environment utilizing SCRUM

Qualifications

10–12 years of overall experience in data engineering and data integration.

Strong architectural and hands-on engineering capabilities.

Experience working on enterprise data modernization initiatives.

Excellent communication and stakeholder management skills.

Ability to work independently in a distributed onshore/offshore delivery model.

Mandatory Skills

Expert-level Python development.

Strong hands-on dbt experience.

Enterprise data ingestion framework development.

Snowflake and AWS S3 experience.

Strong SQL and data integration experience.

Preferred:

Dagster, Control-M, Apache Iceberg, and SnapLogic migration experience.

Vacancy posted 14 hours ago
Similar jobs that could be interesting for youBased on the Sr. Data Engineer in Boston, MA vacancy
  •  ...NessContact: Meghana GorusuCompany: SRI Tech SolutionsJob Title: Sr Data EngineerLocation: Boston, MA (3-4 days a week onsite)Skill Set:...  ...SQL and PythonJob Description:We are looking for a Senior Data Engineer with added emphasis on data analysis and Salesforce knowledge,... 
    Senior
    Work at office
    3 days per week

    SRI Tech

    Boston, MA
    4 days ago
  • Senior Data Engineer Healthcare (Medicare/Medicaid)Position OverviewWe are seeking a Senior Data Engineer to design, build, and operate robust, secure, and scalable data platforms that enable analytics and product teams supporting Medicare and Medicaid programs. This role... 
    Senior

    CyberCoders

    Boston, MA
    1 day ago
  • $150k - $165k

     ...your opportunity to grow professionally in a dynamic, global environment, surrounded by forward-thinking peers.The role of Senior Data Engineer is essential for the successful modernization of our data platforms and practices. The Senior Data Engineer will collaborate... 
    Senior
    Full time

    Goodwin Procter

    Boston, MA
    3 days ago
  • $125.24k - $187.86k

     ...Visa sponsorship/sponsorship assumption and other immigration support are not available for this position. The Senior Data Infrastructure Engineer designs, builds, and scales reliable data platforms that power online transaction processing, analytics, machine learning... 
    Senior
    Full time
    Remote work
    Visa sponsorship

    Lyric

    Boston, MA
    2 days ago
  • $120k - $187k

     ...: Information TechnologyLocation: Boston, MA, United StatesWorkplace: HybridSchedule: Full timePosted: 2026-10-27As a Senior BI Data Engineer, you will design, build, and optimize enterprise-scale data solutions that power analytics and business decision-making across Boston... 
    Senior
    Full time
    Work at office

    The Boston Beer Company

    Boston, MA
    3 days ago
  • $126.2k - $264.1k

     ...days from the posting date or as long as the job remains posted. Career Level - IC5 About Us Only Oracle brings together the data, infrastructure, applications, and expertise to power everything from industry innovations to life-saving care. And with AI embedded... 
    Senior
    Temporary work
    Flexible hours

    Oracle

    Boston, MA
    4 days ago
  • $98.5k - $141.5k

     ...institutions who rely on us to help them build more secure and prosperous futures. THE ROLEThe Investment Data Management Office is seeking a Senior Investment Data Engineer with strong SQL and data quality assurance experience to support the implementation, testing, and... 
    Senior
    Full time
    Work at office
    Local area
    Remote work
    Flexible hours

    MFS Investment Management

    Boston, MA
    4 days ago
  •  ...Embrark Data Strategy Role This is an exciting phase in our growth that has created a unique opportunity to be involved in the...  ...data strategy. You will be working on the Business Intelligence Engineering team and partnering with the Product, Marketing, and Finance teams... 
    Senior

    Software Technology Inc

    Boston, MA
    2 days ago
  • Senior to Principal Data Engineer (SQLserver, Financial Services), Boston, MA. Hybrid, min 3 days week in office, 135k to 175k base on experienceCompensation Commensurate with experience, EOE, diverse highly educated culture Candidates must be a U.S. citizen or national... 
    Senior
    Permanent employment
    Contract work
    Work at office
    Local area
    3 days per week

    BlueSkyClarity

    Boston, MA
    4 days ago
  • $115k - $150k

    About the jobSenior Associate, Data Products | Finance | Boston, MAOverviewJoin a high-performing investment-focused organization as...  ...platforms.Required Experience6+ years of experience in Data Engineering, Data Platform Engineering, or Software Engineering.Deep expertise... 
    Senior
    Work at office
    3 days per week

    Huxley Associates

    Boston, MA
    2 days ago
  • $300k

    Senior Data EngineerOur client is one of the world's largest privately held investment management firms, overseeing hundreds of billions...  ...and best practices across the organizationPartner closely with engineering, data, business, and investment stakeholders to translate... 
    Senior

    Huxley Associates

    Boston, MA
    4 days ago
  •  ...Vice President, Senior Data Engineer About the Company Mission-driven provider of tax credit syndication services Industry Real Estate Type Privately Held Founded 2001 Employees 51-200 Specialties lihtc tax credit syndication low income... 
    Senior

    Confidential

    Boston, MA
    2 days ago
  • Description At Liberty Mutual Investments Technology, we are seeking a skilled Senior Data Engineer who is ready to take on exciting challenges, contribute to our cloud-native data platform, and make a real impact across our investment insights. You will build modern... 
    Senior
    Work at office
    Local area

    Liberty Mutual

    Boston, MA
    2 days ago
  • $140k - $200k

    Boston, MAHybridFull Time$140k - $200k Job Description A rapidly growing scientific data and AI technology company is seeking a Senior Scientific Data Engineer to join its fully remote team. This role offers the opportunity to work at the intersection of data engineering... 
    Senior
    Full time
    Temporary work
    Immediate start
    Remote work

    Motion Recruitment

    Boston, MA
    4 days ago
  • Senior Data Engineer - Boston, United StatesWe are partnering with our client, a leading institutional asset manager based in Boston's financial district, to appoint a Senior Data Engineer to join our client's team. In this permanent role, you will be instrumental in building... 
    Senior
    Permanent employment
    Work at office
    Remote work
    Monday to Friday

    Huxley Associates

    Boston, MA
    4 days ago
  • $171.2k - $228.2k

     ...secure in their home.We’re investing in a multi-quarter Enterprise Data Platform program to move our data and analytics stack onto a...  ...lakehouse. We’re looking for a Senior Manager, Data Platform Engineering to lead the team that builds, operates, and evolves that platform... 
    Senior
    Work at office

    SimpliSafe

    Boston, MA
    3 days ago
  • $195.3k - $199.2k

     .... In this role you will design, develop, and maintain scalable data pipelines and systems using Databricks on Azure (among other environments...  ...sharing capabilities. You will be leading and executing data engineering projects from inception to completion, ensuring timely delivery... 
    Senior
    Apprenticeship
    Easy work

    McKinsey & Company

    Boston, MA
    2 days ago
  • $140k - $160k

     ...company that enables organizations to share and analyze sensitive data at speed without compromising security or control. VIA’s...  ...impressive mission requires an equally impressive Senior Data Analytics Engineer. As a Senior Data Analytics Engineer at VIA, you'll play a... 
    Senior
    Summer work
    Remote work
    Work from home
    Flexible hours

    VIA

    Somerville, MA
    2 days ago
  • Senior Investment Data Engineer, Boston, MA, Hybrid, 2 days in, 150k to 190kCompensation Commensurate with experience, bonus, benefits, EOE, diverse highly educated cultureCandidates must be a U.S. citizen or national, refugee, asylum, or lawful permanent resident. Principals... 
    Senior
    Permanent employment
    Work experience placement
    Casual work
    Work at office

    BlueSkyClarity

    Boston, MA
    4 days ago
  • As a Senior Data Engineer, CASM Platform, you will: Data Integration, API Development: Integrate diverse cybersecurity data sources using variety of API mechanisms and to standardize and streamline data across the data and user planes.Build and maintain data APIs for seamless... 
    Senior
    Contract work

    Cedent Consulting

    Boston, MA
    4 days ago
  • $176k - $179.5k

     ...meaningful impact for patients. You will design and manage scalable data pipelines and secure analytics platforms, lead technical...  ...work in Agile teams alongside data scientists, machine learning engineers, and industry experts to develop impactful analytics solutions... 
    Senior
    Apprenticeship
    Easy work
    Shift work

    McKinsey & Company

    Boston, MA
    2 days ago
  • $140k - $200k

    DescriptionAbout TetraScienceTetraScience is the Scientific Data and AI Company building Tetra OS, the operating system for scientific...  ...TetraScience's forward deployed teams as a senior, hands-on engineer. In this role, you will work directly with biotech, pharmaceutical... 
    Senior
    Full time
    Immediate start
    Flexible hours

    TetraScience

    Boston, MA
    3 days ago
  • $90k - $180k

     ...our perspectives with those of our clients.About the RoleSenior Data EngineerPOSITIONTracing our roots to 1928, we are one of the world...  ...growth, a new opportunity has arisen for a Senior Data Engineer. As a member of the Science Research Technology team, this individual... 
    Senior
    Full time
    Remote work
    Flexible hours
    1 day per week

    Wellington Management

    Boston, MA
    4 days ago
  • $125k - $180k

     ...for the right patients, at the right time.We are building the Patient Evaluation Engine: a high-scale, multi-modal healthcare platform where autonomous AI agents reason over clinical data to drive real-time clinical evaluation across federated networks of hospitals. We... 
    Senior
    Full time
    Remote work

    Tempus

    Boston, MA
    8 hours ago
  • $135k - $141.75k

     ...recognized as one of the industry's best firms to work for. For more information, visitlek.com.About the Role:We are seeking a Senior Data Engineer with deep expertise in data engineering and a strong working command of AI and LLM tech stacks. This role sits within our Data... 
    Senior
    Permanent employment
    H1b
    Local area
    Worldwide
    Home office

    L.E.K. Consulting

    Boston, MA
    4 days ago
  • $120k - $202.5k

    Who we are looking for We are looking for a Senior SIEM Data Engineer reporting directly to the Cyber Data Engineering Manager. You will lead the design, onboarding, transformation, validation, and operational support of cybersecurity telemetry and enterprise log data used... 
    Senior
    Full time
    Temporary work
    Work experience placement
    Work at office
    Local area
    Flexible hours

    State Street Bank

    Quincy, MA
    1 day ago
  •  ...passionate about getting physicians and healthcare providers away from the keyboard and back to clinical care. Overview The Senior Data Engineer is a member of the Data Platform team, reporting to the Director of Machine Learning Engineering and Data. The Data Platform... 
    Senior
    Full time
    Temporary work
    Work at office

    CodaMetrix

    Boston, MA
    8 days ago
  • $165k - $209k

     ...share our passion for impact to join us! Position Summary: We are seeking a highly skilled, hands-on Senior / Principal Data Infrastructure Engineer to design, build, and own the data systems that make our proteomics data discoverable, trustworthy, and usable across the... 
    Senior

    Seres Therapeutics

    Cambridge, MA
    2 days ago
  •  ...Job Description Job Description Job Title: Sr.Data Engineer and Data Engineer Location: CST or EST PS Offfice (Boston, Atlanta, Houston, Chicago, New York) must be 40miles to an office and able to pick up laptop/id verification day 1-3 days a week onsite Length... 
    Senior
    Work at office
    3 days per week
    1 day per week

    Scadea

    Boston, MA
    22 days ago
  • Description The USRM Data Office turns data into a competitive advantage. As a Principal Data Solutions Analyst, you’ll design and...  ...requirements, building prototypes, and partnering with data engineers to deliver scalable data products on Snowflake. You’ll use advanced... 
    Senior
    Work experience placement
    Work at office
    Local area

    Liberty Mutual

    Boston, MA
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Sr. Data Engineer. Be the first to apply!