Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Data Pipeline Engineer/Developer

Gene by Gene

Senior Data Pipeline Engineer/Developer We are seeking a Senior Data Pipeline Engineer/Developer to design, build, and own the data pipelines that move genomic and operational data from lab instruments and LIMS through to analytics, products, and clinical reporting. In this senior individual-contributor role you will set technical direction for our data platform, build production pipelines that are reliable and reproducible at scale, and provide architectural and code-level guidance to existing engineering teams. You will own data quality, lineage, and observability end to end, and operate in a regulated environment where reproducibility and auditability are non-negotiable. This is a Python-first, full-stack engineering role. Accountabilities and Responsibilities Pipeline Engineering & Integration Builds and operates production ETL and ELT for high-volume genomic data as well as operational and business data. Integrates data across LIMS, lab instruments, internal applications, and third-party sources through robust, well-tested interfaces. Develops across the stack in Python (data services, internal APIs, and supporting application code) and provides architectural and code-level guidance to engineering teams on data-layer integration. Architecture & Technical Leadership Sets technical direction for batch and streaming data pipelines by evaluating frameworks, orchestration, storage, and processing patterns, making recommendations, and leading adoption. Models and tunes data stores (Microsoft SQL Server and PostgreSQL, plus a cloud warehouse or lake) for performance and scale. Defines and enforces engineering standards for testing, CI/CD, infrastructure as code, code review, and architecture decision records. Data Quality & Pipeline Observability Owns data quality, lineage, and observability, including freshness, completeness, schema-drift detection, cost-per-job, and SLAs. Builds pipelines for reproducibility and audit-readiness, incorporating versioned data and code, lineage, decision logging, access controls, and evidence collection. Partners with security and compliance on data privacy, PII and PHI handling, and regulatory requirements across the data lifecycle. Regulatory Compliance and Security Governance Apply deep understanding of CAP/CLIA, HIPAA, GDPR, and GxP regulations specifically to data pipeline architecture. Oversee protected health information (PHI) handling, data lineage, retention, and comprehensive audit logging. Produce and maintain the critical operational evidence required to carry the data platform successfully through compliance audits. Adhere to strict data-governance controls necessary for securely handling sensitive genomic data across global regions. Enforce least-privilege access and ensure zero data egress to personal or unapproved infrastructure. Utilize exclusively de-identified or synthetic data within development environments. Maintain strict compliance with international data-residency and localization requirements. Position Requirements Skills and Knowledge Strong SQL proficiency on Microsoft SQL Server and PostgreSQL, including schema design, query tuning, and performance troubleshooting. Strong Python proficiency across the stack (data pipelines, backend services, and APIs). This is a Python-first role. Production experience with a workflow orchestrator (Argo Workflows, Step Functions, Prefect, Dagster, Airflow, or similar). Production experience with containers (Docker) and Kubernetes, which the lead orchestrator (Argo Workflows) runs on. Strong testing discipline, including unit, integration, and data-quality or contract tests. Excellent written and verbal communication, including the ability to explain data tradeoffs to technical and compliance stakeholders. Genomics or NGS data formats and handling (FASTQ, BAM/CRAM, VCF). Bioinformatics workflow engines (Nextflow, WDL/Cromwell, or Snakemake). AWS data stack proficiency (S3, Glue, EMR, Batch, Lambda, Redshift) and/or AWS HealthOmics. Distributed processing and streaming frameworks (Spark, Kafka, Kinesis). Data warehouse and transformation tooling (Snowflake, dbt). Infrastructure as code practices (Terraform, CDK, CloudFormation, or Pulumi). Familiarity with .NET (C#) and/or C++ as supporting languages for integrating with existing services. Knowledge of LIMS integration and on-premises plus hybrid data architectures. Experience 8+ years of professional software or data engineering experience. 4+ years building and operating production data pipelines at scale. 3+ years of production cloud experience (AWS preferred). Demonstrable experience working in regulated environments. Compliance is a hard requirement for this role. Experience supporting HIPAA or GxP audits. Education Bachelor's degree in Computer Science, Bioinformatics, or a related field, or equivalent professional experience. An advanced degree in a quantitative field is preferred. Why Join Us At Gene by Gene, you'll join a mission-driven team advancing the science of genetics and discovery. You'll have the opportunity to shape meaningful campaigns, tell compelling brand stories, and collaborate with talented professionals who share your passion for creativity, curiosity, and impact. Gene by Gene

Vacancy posted 1 day ago
Similar jobs that could be interesting for youBased on the Senior Data Pipeline Engineer/Developer in Houston, TX vacancy
  • Senior Data Engineer - Azure / DatabricksLocation: Houston, TX - HybridPosition Type: Full-TimeA well-established Houston-based company...  ...will build and maintain scalable Lakehouse solutions, develop ELT/ETL pipelines, integrate data from multiple enterprise systems, and... 
    Senior
    Permanent employment
    Work at office

    Technology Recruiting Solutions

    Houston, TX
    1 day ago
  •  ...its clients in their end-to-end Data & AI transformation by combining...  ...types (e.g., H-1B).We are seeking a Senior Data Engineer with strong Energy Trading...  ...Engineering & Cloud Solutions:Design, develop, and optimize cloud-based data pipelines, ETL/ELT workflows, and data... 
    Senior
    H1b
    Visa sponsorship
    Work visa
    3 days per week

    Talan

    Houston, TX
    4 days ago
  • $104k - $120k

     ...experience with agile engineering and problem-solving creativity...  ...across technology, data sciences, consulting,...  ...seeking a hands-on Senior Data Engineer with...  ...scalable batch and streaming pipelines, implement Lakehouse...  ...Databricks Workflows.Develop scalable ingestion and... 
    Senior

    Publicis Media

    Houston, TX
    16 hours ago
  •  ...seeking a hands-on, enterprise-minded Senior Manager, Data Platform & Engineering to lead the technical foundation...  ..., templates, documentation, developer portals, and governed access mechanisms...  ..., data reconciliation, deployment pipelines, environment promotion,... 
    Senior
    For contractors
    Immediate start

    Targa Resources

    Houston, TX
    16 hours ago
  • $124k - $280k

     ...Applied AI Health Services Data Science-Senior Manager you will lead client...  ...focused on designing and developing AI-enabled data solutions that...  ...draw on data and analytics engineering, artificial intelligence,...  ...integration patterns, and data pipelines to support scalable AI... 
    Senior
    Full time
    H1b

    PwC

    Houston, TX
    16 hours ago
  •  ...Technology Centers (ATCs) are the engine for reinvention in our clients...  ...are:An experienced, hands-on Data & AI Engineer with deep hands-...  ..., and optimize scalable data pipelines. You design and implement...  ...Responsibilities:• Architect, develop, and maintain scalable, low-latency... 
    Senior
    Work experience placement
    Live in
    Work at office
    Local area
    3 days per week

    Accenture

    Houston, TX
    3 days ago
  •  ...Technology Centers (ATCs) is the engine for reinvention in our clients...  ...in class, hands -on Snowflake Data & Analytics Engineer with deep...  ...Integration, Data Analytics, Pipeline Development, Agentic AI, RAG,...  ...Search, and Cortex Analyst.• Develop production-grade Retrieval-Augmented... 
    Senior
    Work experience placement
    Live in
    Work at office
    Local area
    3 days per week

    Accenture

    Houston, TX
    1 day ago
  •  ...encourages diverse perspectives, develops our people, and fosters a...  ....Position SummaryThe Senior Platform Engineer will lead the design, operation...  ...of the organization’s data and platform services across...  ....Builds and supports CI/CD pipelines and Azure DevOps workflows... 
    Senior
    Full time
    Local area

    Marathon Petroleum Corporation

    Houston, TX
    1 day ago
  •  ...invest in technology, we are seeking an experienced Senior Director of Software Engineering & Data to lead our Enterprise Applications organization and...  ...ability to maintain critical business systems while developing the capabilities, practices, and solutions that will... 
    Senior
    Full time

    AWC

    Houston, TX
    1 day ago
  •  ...Senior Pipeline Engineer - Transmission & Distribution Houston, Texas, United States The Senior...  ...equipment and construction specifications and data sheets, Requests for Qualifications (...  ...experience to complex projects. Find/develop non-standard design solutions.... 
    Senior
    For contractors

    SolveNow

    Houston, TX
    2 days ago
  •  ...Our client is seeking a Senior Pipeline Engineer to join their team within the Oil & Gas sector, focusing on transmission and distribution projects...  ...technical reports for clients and industry publications Develop and maintain client relationships and identify new business... 
    Senior
    Permanent employment

    Matchtech

    Houston, TX
    4 days ago
  •  ...Senior Pipeline Engineer At EXP, we're driven to provide innovative solutions for the world's built and natural environments. As a team of engineers...  ...direction to junior/intermediate engineers to complete data analysis, drawing and mapping assignments. Interacts with... 
    Senior
    Work at office

    EXP

    Houston, TX
    4 days ago
  •  ...Senior Level Pipeline EngineerDo you want to build an impactful career to change the world for the...  ...opportunity for a Senior Level Pipeline Engineer with at least 8 years of experience to...  ...gas pipeline systems and to further develop a new line of service for existing clients... 
    Senior
    Contract work
    For contractors
    Work at office
    Night shift

    Geosyntec Consultants

    Houston, TX
    16 hours ago
  •  ...Senior Liquids/Gas Pipeline EngineerDNV – Digital & Data Solutions Pipeline Hydraulics Delivery team is seeking a Senior Liquids/Gas Pipeline Engineer. The role will focus on the implementation and support...  ...modeling and simulation projects.Develop, configure, validate, and... 
    Senior
    Full time
    Temporary work
    Work experience placement
    Work at office

    DET Norske Veritas

    Houston, TX
    1 day ago
  • $120k - $146.56k

     ...consulting environmental and engineering firm and one of the top design...  ...stand the test of time. From pipelines to power plants, roadways to...  ...tremendously and is seeking a Senior Pipeline Engineer to join our...  ...equipment specifications and data sheet creation.... 
    Senior
    Full time
    Temporary work
    Part time
    Work at office
    Local area
    Flexible hours

    TRC Companies

    Houston, TX
    3 days ago
  • $166k - $220k

     ...experience with agile engineering and problem-solving creativity...  ...across technology, data sciences, consulting,...  ...experienced, hands-on Senior Architect/Data Engineering...  ...Delivery Architect and develop batch, streaming, and real-time data pipelines. Build and support data... 
    Senior

    Publicis Groupe

    Houston, TX
    14 days ago
  •  ...026-09-14This OpportunityWSP USA Inc is currently seeking a Senior Pipeline Engineerto workout of our office in Chicago, Illinois. The following...  ...Kansas City, MO. The candidate in this role will work with engineering team members across the world to deliver pipeline solutions... 
    Senior
    For contractors
    Work at office
    Local area
    Flexible hours

    WSP Group

    Houston, TX
    3 days ago
  • $100k - $130k

     ...Job Description Job Description Senior Pipeline Engineer – Transmission & Distribution Location: Houston, TX Employment Type: Full-Time,...  ...technical and regulatory compliance. Prepare specifications, data sheets, RFQs, bid summaries, and procurement recommendations... 
    Senior
    Permanent employment
    Full time

    MSR Technology Group

    Houston, TX
    a month ago
  •  ...the development of a robust data ecosystem that transforms drilling...  .... You will work closely with engineers, data scientists, and...  ...scalable batch and streaming data pipelines.Strong knowledge of data...  ...machine learning, or AI, including developing or productionizing analytical... 
    Senior

    National Oilwell Varco

    Houston, TX
    4 days ago
  •  ...communities we serve. Responsibilities The Senior Data Engineer designs, builds, optimizes, and supports scalable...  ...focused environment. Qualifications • Design, develop, and optimize scalable ETL/ELT pipelines integrating data from CRM, OSS/BSS, GIS, network monitoring... 
    Senior

    Ezee Fiber

    Houston, TX
    a month ago
  •  ...OpportunityAccelerate possibilities. The WSP Data Center CoE (Center of Excellence) is...  ...? How to secure power and water? How to develop responsibly and become a valued part of...  ...a search for a  Data Center Mechanical Engineer, Senior Consultant to join our Houston, TX or Dallas... 
    Senior
    For contractors
    Work at office
    Local area
    Flexible hours

    WSP Group

    Houston, TX
    4 days ago
  • Overview We are seeking a hands-on Enterprise Data Architect to lead a data modernization...  ...and build modern data architecture and pipelines, enabling consistent reporting,...  ...Requirements 7+ years in data architecture or data engineering Experience with modern cloud data... 
    Senior

    AMSYS Talent

    Houston, TX
    1 day ago
  •  ...Description We are looking for an experienced Senior Data Engineer to support modernization of the Credit...  ...Credit Risk technology platforms. Develop and maintain complex Oracle PL/SQL...  ...Unity Catalog, and dbt. Develop data pipelines using Java, Python, Scala, and Apache... 
    Senior

    MSys Inc

    Houston, TX
    1 day ago
  • Data Engineer The Data Engineer will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal...  ...include: Create and maintain optimal data pipeline architecture Assemble large, complex data sets that... 
    Senior

    Software Technology Inc

    Houston, TX
    1 day ago
  • Senior Data Engineer - (100% Remote) Fully Remote: Work from anywhere with flexible hours that fit...  ...and implement scalable, efficient data pipelines using Azure Data Lake , Databricks , Snowflake , and Synapse Analytics . Develop and optimize workflows with Apache Spark... 
    Senior
    Remote work
    Flexible hours

    Elios, Inc.

    Houston, TX
    1 day ago
  • Databricks Data Engineer About the Role: We are seeking a highly skilled Databricks Data Engineer to design, build, and optimize our next...  ...capabilities of the Databricks platform—from declarative pipelines to advanced governance—to deliver high-quality data products... 
    Senior

    Alephys

    Houston, TX
    1 day ago
  • Senior Data Engineer We are seeking a Senior Data Engineer to design, build, and operate cloud-scale data platforms and pipelines that support analytics, machine learning, and data products across...  ...principles, and streaming pipelines. Develop software engineering solutions in... 
    Senior

    Peyton Resource Group

    Houston, TX
    1 day ago
  •  ...collective progress.  We are currently recruiting for a Senior Pipeline Design Engineer to join our office in Houston, TX within the Plant...  ...technical bid evaluations. Development and Reporting: Develop personnel skills and flexibility through work assignments... 
    Senior
    Full time
    Contract work
    Temporary work
    Work at office
    Local area
    Flexible hours

    RINA

    Houston, TX
    23 days ago
  •  ...include approximately 50,000 miles of pipelines; 260 million barrels of storage...  ...seeking an experienced Python Software Engineer to join the Big Data and Advanced Analytics department. The...  ...directly with subject matter experts to develop high quality, reliable, scalable,... 
    Senior

    Enterprise Products

    Houston, TX
    16 hours ago
  • $128.4k - $192.5k

     ...the CTO, the Advanced Forward Engineering (AFE) team is comprised of...  ...products. We are seeking a Data Engineer to join AFE delivery...  ...legacy data sources. Ensure data pipelines comply with governance,...  ...the future with confidence and develop answers for the most pressing... 
    Senior
    Summer holiday
    Work at office
    Immediate start
    Flexible hours

    EY

    Houston, TX
    6 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Data Pipeline Engineer/Developer. Be the first to apply!