Senior Software Engineer

Senior Software Engineer at Datadog. About the role The Deployment Gates team is responsible for creating systems that evaluate the safety of software before it is released into production environments. In this role, you will be tasked with developing AI-enhanced tools that seamlessly integrate into CI/CD pipelines, allowing for the analysis of observability data to automate deployment decisions. This position requires a blend of product strategy and backend engineering expertise to empower customers to avert incidents proactively. Key facts Location: Madrid, Spain Engagement: Hybrid Team: Deployment Gates What you’ll do Design and implement decision-making systems that assess customer deployments by leveraging CI/CD context and telemetry data to deliver clear allow or block outcomes. Establish and execute evaluation metrics for AI performance, focusing on precision and recall, while overseeing shadow mode validation processes. Create backend services that are integral to customer deployments and participate in on-call rotations to ensure system reliability. Collaborate closely with product management to convert user insights and roadmap goals into actionable technical specifications. Work alongside internal teams dedicated to Faulty Deployment Detection, LLMs, and AI agents to enhance system capabilities. Develop safety mechanisms, including conservative defaults and human-in-the-loop processes, to maintain high system reliability. Engage in continuous improvement of deployment processes through feedback loops and iterative development. Analyze and optimize the performance of backend systems to ensure low latency and high resiliency during deployment cycles. Requirements A minimum of 5 years of experience in backend systems, with a strong emphasis on microservices performance, concurrency, latency, and resiliency. Demonstrated success in delivering software features that utilize LLMs or AI agents in production settings. Proficiency in a contemporary programming language, along with substantial experience in API and service design. Familiarity with managing the entire product lifecycle, from initial prototype development to general availability. Capability to conduct both offline and online evaluations for AI systems to ensure effectiveness and reliability. Experience utilizing AI coding tools to validate and enhance generated outputs effectively. Strong understanding of production operations, including monitoring systems and alerting mechanisms. Nice to have Experience with Continuous Delivery tools such as ArgoCD, Argo Rollouts, Spinnaker, or Octopus Deploy, which can enhance deployment processes. Knowledge of agent frameworks, retrieval-augmented generation (RAG), tool-use orchestration, and indexing techniques for observability data. Familiarity with cloud platforms and their integration into deployment pipelines can be an added advantage. Skills & tools Expertise in backend microservices architecture Implementation of LLMs and AI agents Integration with CI/CD pipelines Proficient in observability and telemetry practices Experience in production monitoring and on-call operations Strong API design skills Practical notes The compensation package includes stock equity options (RSUs) for new hires and an employee stock purchase plan (ESPP) to encourage long-term investment in the company. Employees and their dependents aged 6 and older benefit from global mental health support services. Datadog is committed to professional development, offering mentorship programs and access to employee resource groups to foster a supportive work environment. Benefits and opportunities for growth may differ based on the specifics of your employment and geographical location. In this role, you will have the chance to work in a dynamic environment where innovation and collaboration are at the forefront. You will be part of a team that values your input and encourages you to take initiative in developing solutions that enhance the deployment process for our customers. If you are passionate about backend engineering and AI technologies, this position offers an exciting opportunity to make a significant impact in the field. #J-18808-Ljbffr