Principal Software Engineer, Distributed Systems

Job Description

Background \n

\n

We are hiring a Staff / Principal Site Reliability Engineer focused on distributed systems and platform reliability . This is not a traditional DevOps role.

\n

\n

You will own the architecture and engineering practices that ensure Sigma workflows execute reliably even when individual components fail.

\n

\n

Processes crash. Networks become unavailable. APIs time out. Workers hang. Messages may be delivered more than once. Databases experience contention. Kubernetes pods restart.

\n

\n

Your job is to design Sigma so those conditions are expected, detected, and automatically recovered from.

\n

The fundamental principle is simple:

\n

\n

An accepted Sigma job may fail, but it should never disappear.

\n

\n

You will work closely with engineering leadership to evolve Sigma's execution architecture, improve production reliability, establish observability standards, and eliminate classes of distributed-system failures before they affect customers.

What You'll Own\n

\n

Design and improve the systems responsible for executing long-running and scheduled infrastructure workflows.

\n

Build patterns for:

\n

\n
• Durable workflow execution
\n
• Idempotent operations
\n
• Automatic retries and exponential backoff
\n
• Worker leases and heartbeats
\n
• Stuck-job detection and recovery
\n
• Workflow timeouts and cancellation
\n
• Failure isolation
\n
• Dead-letter handling
\n
• Checkpointing and workflow resumption
\n
• Concurrency control
\n
• Distributed locking and lease management
\n
• Reconciliation loops
\n
• Graceful degradation during dependency failures
\n

\n

Evaluate and implement workflow technologies such as Temporal or equivalent systems where appropriate.

\n

Messaging and Event Infrastructure\n
\n

Own and improve Sigma's asynchronous execution architecture.

\n

Work extensively with technologies such as:

\n

\n
• Kafka
\n
• Distributed consumers
\n
• Consumer groups
\n
• Message delivery semantics
\n
• Partitioning
\n
• Consumer lag
\n
• Backpressure
\n
• Retry strategies
\n
• Event ordering
\n
• Duplicate message handling
\n
• Poison-message isolation
\n

\n

Ensure event-processing failures cannot silently cause workflows to stop progressing.

\n

Database Reliability\n
\n

Help design application patterns that remain reliable under high concurrency.

\n

Deeply understand and troubleshoot:

\n

\n
• PostgreSQL transactions
\n
• Row and advisory locking
\n
• Deadlocks
\n
• Lock contention
\n
• Transaction isolation
\n
• Connection pooling
\n
• Long-running transactions
\n
• Query performance
\n
• Database failover
\n
• Schema and migration safety
\n

\n

Partner with application engineers to identify and eliminate patterns that create production contention or reliability risks.

\n

Kubernetes and Platform Reliability\n
\n

Improve the reliability of Sigma deployments running in containerized environments.

\n

Responsibilities include:

\n

\n
• Kubernetes workload architecture
\n
• Pod lifecycle and failure recovery
\n
• Resource limits and capacity planning
\n
• Autoscaling
\n
• Health checks
\n
• Graceful shutdown
\n
• Rolling deployments
\n
• High availability
\n
• Infrastructure-as-code
\n
• Disaster recovery
\n
• Backup and restore validation
\n

\n

Design systems that tolerate node, pod, process, and dependency failures without losing customer work.

\n

Observability\n
\n

Make the state of Sigma understandable in real time.

\n

Build and improve:

\n

\n
• Metrics
\n
• Distributed tracing
\n
• Structured logging
\n
• Dashboards
\n
• Alerting
\n
• Workflow-level telemetry
\n
• Kafka consumer monitoring
\n
• Database performance monitoring
\n
• Worker health monitoring
\n
• Queue depth and backlog monitoring
\n

\n

Technologies may include:

\n

\n
• OpenTelemetry
\n
• Grafana
\n
• Prometheus
\n
• Sentry
\n
• CloudWatch
\n

\n

Engineers should be able to answer questions such as:

\n

\n
• Why is this workflow still running?
\n
• Which worker owns this task?
\n
• When did it last make progress?
\n
• Which dependency is causing the delay?
\n
• Did this operation execute once or multiple times?
\n
• Can the workflow safely resume?
\n
• Are scheduled jobs starting when expected?
\n

\n Reliability Engineering\n
\n

Help establish engineering practices expected of mature enterprise platforms.

\n

This includes:

\n

\n
• Service-level objectives and indicators
\n
• Error budgets
\n
• Capacity planning
\n
• Load testing
\n
• Failure-mode testing
\n
• Chaos testing
\n
• Incident response
\n
• Root-cause analysis
\n
• Production readiness reviews
\n
• Runbooks
\n
• Automated recovery
\n
• Disaster-recovery testing
\n

\n

Move Sigma from detecting incidents to preventing and automatically recovering from them.

\n

What We're Looking For\n
\n

You have significant experience operating production distributed systems where reliability matters.

\n

Strong candidates will have deep experience with several of the following:

\n

\n
• Kubernetes
\n
• Kafka or similar event-streaming systems
\n
• PostgreSQL
\n
• Distributed systems
\n
• Asynchronous worker architectures
\n
• Workflow orchestration
\n
• Temporal, Cadence, Step Functions, Conductor, or similar systems
\n
• Python backend systems
\n
• Cloud infrastructure
\n
• Infrastructure-as-code
\n
• Observability platforms
\n
• High-availability architectures
\n

\n

You understand concepts such as:

\n

\n
• At-least-once delivery
\n
• Idempotency
\n
• Leases and fencing tokens
\n
• Distributed locks
\n
• Leader election
\n
• Retry semantics
\n
• Backpressure
\n
• Eventual consistency
\n
• Transaction boundaries
\n
• Failure domains
\n
• Reconciliation
\n
• Split-brain scenarios
\n
• Distributed tracing
\n

\n

Most importantly, you have personally diagnosed and improved production systems experiencing real distributed-system failures.

\n

What Success Looks Like\n
\n

Within your first several months, you will help Sigma establish an architecture where:

\n

\n
• Scheduled workflows reliably begin when expected
\n
• Accepted work cannot silently disappear
\n
• Worker crashes do not result in lost workflows
\n
• Infrastructure failures automatically recover where possible
\n
• Long-running workflows can safely resume
\n
• Duplicate execution is prevented or safely handled
\n
• Stuck workflows are automatically detected
\n
• Database and messaging contention is visible before becoming an outage
\n
• Engineers can trace a workflow from request through every execution step
\n
• Customer environments can scale without requiring manual infrastructure babysitting
\n