Staff Software Engineer, Observability/Site Reliability

PsiQuantum is building fault-tolerant quantum computers using silicon photonics. The Observability team builds and operates the telemetry platform that monitors quantum computing systems, on-premises Kubernetes clusters, cloud infrastructure, and custom hardware including control electronics and cryostats. This Staff-level role combines hands-on observability engineering with technical leadership, owning major parts of the stack from design through operation.

What You'll Do

• Design, deploy, and operate observability platform across on-premises Kubernetes and cloud, covering collection, storage, querying, visualization, and alerting for metrics, logs, and traces
• Roll out OpenTelemetry across the fleet: deploy and configure Collectors, design telemetry pipelines, and help service teams adopt OTel SDKs in Python, Rust, and other languages
• Run Prometheus in production including service discovery, scrape configuration, recording and alerting rules, Alertmanager routing, retention, high availability, and long-term storage
• Work with hardware teams to instrument custom devices, control electronics, and cryostats; build exporters and collectors for device telemetry
• Define and publish telemetry standards for metric naming, cardinality budgets, structured logging, and trace context propagation
• Participate in on-call rotation, respond to incidents, diagnose production issues, and lead blameless post-incident reviews
• Mentor engineers, lead design reviews, and coordinate observability needs across software, hardware, research, and operations teams
• Use AI and LLM tools responsibly to accelerate engineering work including code generation, query building, and incident analysis

What You Need

• 8+ years of professional software, infrastructure, or site reliability engineering experience, including significant time building and operating observability systems in production
• Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, Mathematics, Physics, or closely related technical discipline
• Deep hands-on experience with Prometheus and Grafana, including PromQL, recording and alerting rules, Alertmanager, exporters, and dashboard design at scale
• Production experience with OpenTelemetry, including Collector, OTLP, SDK, automatic instrumentation, and semantic conventions
• Hands-on experience running workloads on self-managed on-premises Kubernetes and cloud with containers, Helm, and infrastructure as code such as Terraform
• Strong programming skills in Python and at least one of Go or Rust, with ability to write production-quality exporters, Collector components, and automation
• Solid knowledge of Linux and networking fundamentals including processes, filesystems, resource limits, DNS, TCP/IP, and TLS
• Experience carrying a pager for production systems, leading incident response, running post-incident reviews, and designing actionable alerts
• Staff-level technical leadership: leading design work, setting standards, mentoring engineers, influencing across teams, and communicating tradeoffs
• Proficiency with modern AI/LLM engineering tools such as coding copilots, model APIs, and retrieval workflows, with disciplined approach to security and code review

Nice to Have

• Experience with scalable metrics, log, and trace backends such as Thanos, Mimir, VictoriaMetrics, Loki, Tempo, OpenSearch, or ClickHouse
• Experience collecting telemetry from laboratory or industrial equipment over protocols such as Modbus, OPC UA, SNMP, SCPI, or serial interfaces
• Familiarity with cryogenic, vacuum, laser, or other lab systems, or with software used in laboratories, data centers, or advanced manufacturing
• Experience with high-frequency or high-cardinality time-series data, including downsampling and data lifecycle management
• Experience running observability across multiple sites with hybrid on-premises and cloud environments including constrained network connectivity
• Experience with eBPF-based observability, continuous profiling, or Kubernetes operators
• Experience with SLO tooling, synthetic monitoring, or incident management platforms such as PagerDuty
• Familiarity with Kafka, AWS, or VAST Data
• Contributions to OpenTelemetry, Prometheus, Grafana, or related open-source projects
• Background in physics or quantum computing is helpful but not required