Senior Data Engineer

Senior Data Engineer We are seeking a Senior Data Engineer to design, build, and operate cloud-scale data platforms and pipelines that support analytics, machine learning, and data products across the organization. This role collaborates with Product, Architecture, and Data Science teams to deliver secure, reliable, and cost-efficient batch and streaming data services. The ideal candidate brings deep technical expertise in cloud data engineering, data platform architecture, and best practices for governance, reliability, and observability. Key Responsibilities Lead the design, development, and operation of enterprise data platforms, including lakehouse, streaming, and centralized data hubs, while managing dependencies and integrations with enterprise systems. Provide technical leadership and guidance to engineering teams, including planning, scoping, risk management, and delivery accountability. Coach engineers on standards, observability, operational excellence, and best practices; conduct design and code reviews. Define and evolve cloud data architectures for ingestion, storage, compute, cataloging, and access across real-time and batch use cases. Build and maintain robust ELT/ETL and streaming pipelines using orchestration frameworks (Airflow, Dagster), event platforms (Kafka, Kinesis, Event Hub), and distributed processing frameworks (Spark, Databricks, Flink, Beam). Implement lakehouse/lake patterns with medallion layering, CDC, and schema evolution; ensure quality gates, testing, and lineage. Establish and monitor service-level objectives for data freshness, availability, and quality; implement observability and runbooks for operations. Optimize storage and compute costs and performance; manage capacity, autoscaling, and governance. Ensure privacy, security, and compliance (IAM, tokenization, row/column security, data retention); work closely with enterprise security teams. Translate product and stakeholder requirements into data contracts and SLAs; document architecture and operational procedures. Evaluate emerging data technologies and provide proof-of-concepts and reference implementations. Anticipate future platform needs, including multi-cloud, IoT/edge ingestion, and AI/ML workloads. Modernize existing data architectures using current best practices: lakehouse, CDC, medallion layering, data mesh principles, and streaming pipelines. Develop software engineering solutions in Python/SQL and one additional language (Scala/Java); familiarity with REST/gRPC and microservices patterns. Utilize cloud data services (Azure Data Lake/Databricks/Synapse; AWS S3/EMR/Glue/Redshift; GCP BigQuery/Dataflow) and distributed stores (Delta/Iceberg/Hudi, Kafka, Redis). Work with SQL/NoSQL databases (Postgres, DynamoDB, MongoDB, Cassandra) and CI/CD pipelines, containerization (Docker/Kubernetes), and infrastructure as code (Terraform). Ensure proper data governance and security across catalogs, lineage, RBAC/ABAC, encryption, and key management. Communicate effectively with stakeholders, translating business needs into scalable, reliable data solutions. Required Qualifications Bachelor's degree in Computer Science, Engineering, or related field; Master's preferred. 5–10+ years of experience in data engineering or platform roles delivering enterprise-grade systems (SaaS or large-scale internal platforms). 2+ years leading or architecting cloud data platforms at scale. Proven experience with batch and streaming pipelines, lakehouse implementations, and production ML data flows. Experience operating and supporting data platforms with SLAs/SLOs, cost governance, and on-call responsibilities. Hands-on experience with cloud-native data services, orchestration frameworks, and distributed processing. Preferred / Additional Experience Experience with real-time data systems and IoT ingestion. Background in large-scale SaaS or multi-cloud platforms. Strong collaboration skills with cross-functional teams, including product, architecture, and analytics. Peyton Resource Group