Sr. Inference Service SRE & Automation Engineer

Bitdeer is a world-leading technology company for Bitcoin mining and AI cloud.

NeoCloud runs Infermesh, our managed Inference / Model-as-a-Service platform: customers call model endpoints over a public API and pay per token. For a service like this, reliability is the product — a failed request is not an internal incident, it is a customer-visible SLA breach. In this role you own the reliability of the inference serving path end to end: public API gateway, inference router, request scheduler and queue, prefill/decode and KV cache, inference engines, and the GPU model replicas behind them — plus the control plane and shared dependencies they rely on. Your mandate is to aim a 99.95% SLA, by closing the architecture, automation, and operational gaps that stand in the way. You are the SRE who stands between a fleet of H200/B300/GB300/VR NVL72 serving nodes and the customer's experience of a simple, dependable model endpoint.

• End-to-end availability of customer inference endpoints — API availability, request success rate, TTFT/TPOT latency, and the published SLA.
• The serving path: API gateway, inference router, health-based routing, request scheduler, queueing and admission control, prefill/decode serving, and the KV cache service.
• Inference runtime operations: engine health (vLLM / SGLang / TensorRT-LLM class runtimes), model load and readiness validation, model revision rollout and rollback.
• Model replica topology and failure isolation: tensor-parallel group containment, GPU / NVLink / NCCL fault isolation, replica anti-affinity across nodes, racks, and failure domains.
• Multi-cluster serving across MY / SG / JP: cross-cluster traffic steering, tested cluster failover, and regional recovery or evacuation scope.
• GPU serving fleet health (H200 / B300 / GB300 / VR NVL72 bare metal): DCGM / Xid / ECC monitoring, automated drain and cordon, replica replacement with no sustained customer impact.
• The Kubernetes platform beneath the serving path: HA control plane, etcd, CNI, CoreDNS, public / management / east-west edge tiers, and egress NAT.
• Shared control-plane dependencies: PostgreSQL (including Temporal), Redis, Vault secrets and certificates, object storage and model registry, ClickHouse — HA, tested failover, and decoupling live serving from control-plane outages.
• Capacity and saturation engineering: GPU utilization, KV cache occupancy, queue depth and concurrency; N+1 headroom, admission control, load shedding, and graceful degradation instead of collapse.
• The SLI/SLO/SLA system: define and instrument the SLIs, publish the SLOs, run error budgets and burn-rate alerting, and produce customer-facing SLA reporting.
• Observability stack — VictoriaMetrics, VictoriaLogs, Tempo, Pyroscope, Grafana, ClickHouse — plus external synthetic inference probes that exercise the real customer API path.
• Incident management: severity model, incident commander rotation, primary / secondary / cross-team / vendor escalation, status-page and customer communication, and postmortems with tracked actions.
• Change safety: staged and canary rollout for application, model, and configuration changes; automatic rollback on regression; production change review and release risk classification.
• Resilience validation: recurring failover drills, DR tests, game days and fault injection, and backup restore verification — with measured RTO as the evidence.

• You are accountable for the customer's reliability experience — when an endpoint degrades, a replica dies, or a router loses a backend, you own detection, remediation, and the communication loop.
• Drive the platform from to 99.95%, using the SLA gap assessment — architecture, operations, and failure-mode matrices — as your roadmap.
• Own the failure-mode and resilience matrix: every Critical and High failure mode gets a detection signal, a mitigation path, a target RTO, and a drill that proves it.
• Define and publish customer-facing SLOs and SLAs, and drive error-budget-based prioritization between feature work and reliability work.
• Partner with support and customer success to close the loop from customer-reported issues to systemic fixes, and build tenant-visible status, usage, and quota observability that reduces support load.

• The remediation actuator and workflow engine land here — you make the serving control plane safe for automated action.
• Your runbooks, CRDs, and workflow definitions are the schema the platform's predictors and remediators write against.
• Every human intervention you do this quarter becomes an autonomous workflow next quarter — turning customer-impacting incidents into self-healing events.

• Every Critical / High failure mode in the resilience matrix has tested detection, an automated or documented mitigation, and a measured RTO.
• Cross-cluster inference failover drilled and completing inside the SLA target; no single-cluster or shared-component SPOF left unidentified.
• Automatic rollback in place for bad application, model, and configuration deployments — a known-bad canary is reverted without human action.
• External synthetic inference probing and SLO dashboards live; error-budget burn thresholds trigger documented action.
• MTTD and MTTR for customer-impacting incidents measurably reduced, with the top recurring interventions converted into automated workflows.
• On-call rotation with no single-person dependency, and P1/P2 runbooks an on-call engineer can execute without the original author.

• 5+ years in SRE / production platform operations, including 2+ years on a customer-facing API service operating against a published SLA.
• Deep Kubernetes operations experience: HA control plane, etcd, CNI, service discovery, ingress and edge tiers, and multi-cluster topologies.
• Hands-on production experience serving LLM or other model inference — vLLM, SGLang, TensorRT-LLM, Triton, or equivalent: model loading, batching, KV cache behaviour, concurrency limits, TTFT/TPOT latency.
• GPU fleet operations: DCGM / Xid / ECC signal triage, NVLink and NCCL failure diagnosis, node drain and replica recovery.
• Traffic engineering for an availability-critical API: L4/L7 load balancing, gateways, active health checking, DNS/GSLB, and cross-cluster or cross-region failover.
• Strong SRE fundamentals: SLI/SLO/SLA design, error budgets, incident command, postmortems, and capacity planning.
• Operating stateful dependencies in production: PostgreSQL replication and failover, Redis, message queues, object storage, and secrets / certificate lifecycle.
• Observability at scale: Prometheus- or VictoriaMetrics-class TSDB, Grafana, log and trace pipelines, synthetic probing, and alert quality management.
• Overload protection in practice: admission control, concurrency and queue limits, load shedding, and graceful degradation.
• Proficiency in Terraform, Helm, and GitOps workflows (ArgoCD/Flux).
• Strong programming skills in Go or Python for automation, controllers, and operator development.
• AIOps aptitude — you view the control plane as an execution surface for automated remediation, not just a scheduler.
• Runbook-as-code mindset — every SRE playbook you write should be executable by the platform.
• Comfortable with customer-facing incident communication and SLA reporting.

• A culture that values authenticity and diversity of thoughts and backgrounds;
• An inclusive and respectable environment with open workspaces and exciting start-up spirit;
• Fast-growing company with the chance to network with industrial pioneers and enthusiasts;
• Ability to contribute directly and make an impact on the future of the digital asset industry;
• Involvement in new projects, developing processes/systems;
• Personal accountability, autonomy, fast growth, and learning opportunities;
• Attractive welfare benefits and developmental opportunities such as training and mentoring.

--------------------------------------------------------------------

#LI-ST1