Software Engineer III

Responsibilities

• Implement resilient batch, streaming, and CDC ingestion from AWS sources into Databricks.
• Build maintainable data pipelines using Delta Live Tables or Databricks Jobs with testing and documentation.
• Productionize workloads through Databricks Workflows and Jobs with retries, checkpointing, idempotency, and safe reruns.
• Implement data governance and security controls covering least privilege, PII classification, auditing, lineage, metadata, and controlled data sharing.
• Tune Spark and Delta workloads, right-size clusters, optimize storage layouts, and manage job and warehouse costs.
• Use Terraform for Databricks and AWS infrastructure and promote resources across environments.
• Create unit and integration tests, data quality checks, contract tests, replay procedures, and backfill procedures.
• Maintain monitoring and alerting, participate in incident response and root-cause analysis, and manage SLOs and SLAs.
• Use enterprise-authorized AI-assisted development tools for coding, testing, troubleshooting, and documentation while validating generated outputs.
• Maintain version control, code reviews, documentation, and operational runbooks.

Requirements

• Formal training or certification in software engineering concepts and at least 3 years of hands-on software development lifecycle experience.
• Experience building and maintaining data pipelines.
• Hands-on experience building and operating a Databricks Lakehouse hosted in AWS.
• Experience with Delta Lake, including ACID tables, partitioning, and schema evolution.
• Proven experience with Spark on Databricks, including performance tuning, cluster sizing, skew mitigation, joins, caching, and file sizing.
• Experience with streaming and batch pipelines, Structured Streaming, incremental processing, backfills, and late-arriving data.
• Strong AWS fundamentals for data platforms, including S3, IAM, KMS, VPCs, security groups, logging, and auditing.
• Experience implementing Databricks data governance and security controls such as Unity Catalog, table and column permissions, and credential passthrough patterns.
• Experience with monitoring, alerting, incident response, root-cause analysis, and SLO/SLA management.
• Understanding of responsible AI use in engineering workflows, including data sensitivity, secure input and output handling, resiliency, and security expectations.
• Hands-on experience with enterprise-authorized AI-assisted software development tools and the ability to evaluate, validate, and refine their outputs for correctness, performance, and security.
• Preferred: familiarity with modern front-end technologies and exposure to cloud technologies.

Benefits

• Competitive total rewards and benefits package, with additional compensation details provided during the hiring process.
• Eligible roles may receive commission-based pay or discretionary incentive compensation in cash and/or forfeitable equity.
• Benefits may include comprehensive health care coverage, on-site health and wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching.