Principal Software Engineer, Applied AI - Splunk Observability Cloud
$241.4k - $322.6kCISCO Systems
The application window is expected to close on: 11/29/2026Meet the TeamThe Observability Cloud AI team builds the intelligent AI SRE experience for Splunk Observability Cloud, including an AI Assistant and Troubleshooting Agent that investigate production problems across metrics, traces, logs, and infrastructure. We operate within one of the industry's largest observability data estates, enabling us to deliver autonomous detection, incident grouping, and code-level remediation. The team is composed of dedicated engineers focused on applied AI who utilize a "customer-zero" methodology, deploying to our own internal SRE teams before any public release. This collaborative and fast-paced environment prioritizes rigorous evaluation over prototypes, ensuring that every contribution directly improves the reliability of our customers' complex digital systems.Your ImpactAs the Principal Engineer, you will define the technical trajectory for agentic investigation, ensuring our AI solutions achieve production-grade trust and efficiency. You will architect the critical evaluation systems and retrieval layers that allow models to reason effectively over vast telemetry datasets, directly improving accuracy and time-to-answer for incident resolution. You will establish the production standards for our AI agent's latency, cost, and security, providing the necessary guardrails for safe, autonomous remediation. By mentoring team members and partnering with chief architects, you will raise the applied-AI capabilities across the observability portfolio and set the technical bar for the entire organization. You will successfully drive long-term technical direction for these systems, effectively bridging the gap between innovative AI research and mission-critical product features.Minimum Qualifications12+ years of professional software engineering experience.3+ years of experience designing, building, and deploying production-grade LLM-based systems, including at least one agentic system with active tool use.Experience developing evaluation frameworks for AI/ML systems, including dataset creation, offline/online scoring, and release validation.Experience engineering retrieval and context pipelines for structured and semi-structured data, such as telemetry, logs, or time-series databases.Knowledge of production-grade engineering principles including latency optimization, cost management, multi-tenancy, and security guardrails for AI.Preferred QualificationsExperience with Observability, SRE, or incident management domains, including distributed tracing and alerting workflows.Experience working with cloud-native infrastructure at scale, specifically Kubernetes.History of active open-source contributions in AI, agent frameworks, or observability ecosystems (e.g., OpenTelemetry, MCP).Experience leading customer-zero or field-deployed programs to gather actionable feedback for product improvements.Strong communication skills, with a track record of setting technical direction and mentoring engineers through design reviews and technical documentation.Why Cisco? At Cisco, we’re revolutionizing how data and infrastructure connect and protect organizations in the AI era – and beyond. We’ve been innovating fearlessly for 40 years to create solutions that power how humans and technology work together across the physical and digital worlds. These solutions provide customers with unparalleled security, visibility, and insights across the entire digital footprint.Fueled by the depth and breadth of our technology, we experiment and create meaningful solutions. Add to that our worldwide network of doers and experts, and you’ll see that the opportunities to grow and build are limitless. We work as a team, collaborating with empathy to make really big things happen on a global scale. Because our solutions are everywhere, our impact is everywhere. We are Cisco, and our power starts with you. Message to applicants applying to work in the U.S. and/or Canada:The starting salary range posted for this position is $241,400.00 to $322,600.00 and reflects the projected salary range for new hires in this position in U.S. and/or Canada locations, not including incentive compensation*, equity, or benefits.Individual pay is determined by the candidate's hiring location, market conditions, job-related skillset, experience, qualifications, education, certifications, and/or training. The full salary range for certain locations is listed below. For locations not listed below, the recruiter can share more details about compensation for the role in your location during the hiring process.U.S. employees are offered benefits, subject to Cisco’s plan eligibility rules, which include medical, dental and vision insurance, a 401(k) plan with a Cisco matching contribution, paid parental leave, short and long-term disability coverage, and basic life insurance. Please see the Cisco careers site to discover more benefits and perks. Employees may be eligible to receive grants of Cisco restricted stock units, which vest following continued employment with Cisco for defined periods of time.U.S. employees are eligible for paid time away as described below, subject to Cisco’s policies:10 paid holidays per full calendar year, plus 1 floating holiday for non-exempt employees1 paid day off for employee’s birthday, paid year-end holiday shutdown, and 4 paid days off for personal wellness determined by CiscoNon-exempt employees** receive 16 days of paid vacation time per full calendar year, accrued at rate of 4.92 hours per pay period for full-time employeesExempt employees participate in Cisco’s flexible vacation time off program, which has no defined limit on how much vacation time eligible employees may use (subject to availability and some business limitations)80 hours of sick time off provided on hire date and each January 1st thereafter, and up to 80 hours of unused sick time carried forward from one calendar year to the nextAdditional paid time away may be requested to deal with critical or emergency issues for family membersOptional 10 paid days per full calendar year to volunteerFor non-sales roles, employees are also eligible to earn annual bonuses subject to Cisco’s policies.Employees on sales plans earn performance-based incentive pay on top of their base salary, which is split between quota and non-quota components, subject to the applicable Cisco plan. For quota-based incentive pay, Cisco typically pays as follows:.75% of incentive target for each 1% of revenue attainment up to 50% of quota;1.5% of incentive target for each 1% of attainment between 50% and 75%;1% of incentive target for each 1% of attainment between 75% and 100%; andOnce performance exceeds 100% attainment, incentive rates are at or above 1% for each 1% of attainment with no cap on incentive compensation.For non-quota-based sales performance elements such as strategic sales objectives, Cisco may pay 0% up to 125% of target. Cisco sales plans do not have a minimum threshold of performance for sales incentive compensation to be paid.The applicable full salary ranges for this position, by specific state, are listed below:New York City Metro Area:$250,600.00 - $416,900.00Non-Metro New York state & Washington state:$241,400.00 - $371,000.00* For quota-based sales roles on Cisco’s sales plan, the ranges provided in this posting include base pay and sales target incentive compensation combined.** Employees in Illinois, whether exempt or non-exempt, will participate in a unique time off program to meet local requirements.Job SummaryJob number: 2028421Profession: Product and Engineering
$104.5k - $234.6k
...practices. Drives observability standards, capacity... ...Platform Software Development: Lead... ...guidance and coaching to engineers to drive... ...best practices and applies them to software development... ...on a public cloud platform (e.g., AWS... ...saving care. And with AI embedded across...CloudTemporary workFlexible hoursShift work$139.3k - $203.6k
...Meet the Team Join Splunk’s Framework &... ...that power Splunk Cloud. Our team focuses... ...making it easier for engineering teams to build, deploy... ...As a Senior Software Reliability Engineer... ...engineers, and explore AI-assisted... ...Message to applicants applying to work in the U.S...CloudSplunkFull timeTemporary workLocal areaRemote workFlexible hours$87.1k - $157.45k
...perimeter boundaries, multi-cloud platforms, and... ...Adversary Campaign Tracking: Apply the MITRE ATT&CK and... ...defenders and boundary engineers to preempt emerging cyber... ...into enterprise SIEMs (Splunk, Microsoft Sentinel,... ...extraction). * Familiarity with AI prompting (Gemini, Grok...CloudSplunkContract workLocal areaImmediate startShift work$220k - $275k
...records to powering the AI revolution in... ...in healthcare. Staff Software Engineer The Role As a Staff... ...comfortably within our multi-cloud, large-scale... ...system integrations Observability: Enterprise-level telemetry... ...level are expected to apply deep architectural...Cloud$113k - $254.2k
...Robinson is seeking a Senior Software Engineer on our Book-It-Now/... ...and correctness of AI and Human pricing and... ...performance, data quality, observability, and operational... ...building and operating cloud-native applications Experience... ...candidates to apply at their earliest...CloudHourly payFull timeContract workTemporary workRemote workWorldwideFlexible hours$146.4k - $263.6k
...building cutting-edge AI systems? Are you ready... ...the Akamai Inference Cloud Team! The Akamai Inference... ...best As a Senior II Software Engineer Lead, you will be... ...acceleration Have knowledge of observability, monitoring, and... ...requirements apply. Equal Employment Opportunity...CloudWork experience placementWork at office$233.3k - $385k
...customer-facing Fellow AI Engineer to define and lead UKG’s... ...standards for scalability, observability, cost optimization, and performance across cloud-native deployments. Innovation & Applied Research Stay at the... ...Experience: 15+ years in software development and AI, with...CloudLocal area$169.8k - $284.3k
...Opportunity Oracle Cloud Infrastructure is seeking a Master Principal or Distinguished... ...wins. As enterprise AI moves into... ...platform teams, and engineers from discovery through... ..., edge, security, observability, and resilient... ...distributed systems who can apply that experience to...CloudTemporary workFlexible hours- ...7924 Job Title: Staff Software Development Engineer About Trellix Trellix... ...build cyber resilience for the AI era. Combining AI-native... ...technology, hybrid, and cloud environments. Trellix helps... ...data residency guidelines. Observability & Telemetry: Design...CloudFull timeWork at officeRemote workFlexible hours3 days per week
$111.3k - $215.1k
...Lead, Forward Deployed Engineering to join our AI & Data Labs practice.... ...standard for secure, observable, and auditable code... ...end-to-end production software systems, with a focus... ...experience with cloud platforms (Azure, GCP... ...are considered as they apply, until the opportunity...CloudH1bLocal area- ...You will deepen customer relationships and apply knowledge of ML and predictive analytics... ...sales experience selling enterprise software to large enterprise customers ~ Experience... ...security, analytics Subscription, SaaS, or cloud software experience Knowledge of...CloudSplunkRemote work
$115.5k - $180.53k
...with cybersecurity, network engineering, infrastructure, and program... ...Virtualization (NFV), SD-WAN, and cloud-based networking concepts.... ...tools such as Power BI, Tableau, Splunk, ELK, or similar platforms.... ...'s toughest problems and apply advanced solutions to our clients...CloudSplunkPermanent employmentFull timeContract workPart timeLocal areaRemote work$150.29k - $225.43k
...Lyric combines responsible AI with clinical, payment, regulatory... ...payments. The Staff Software Engineer drives the design and implementation... ...(SQL and NoSQL), as well as cloud infrastructure (e.g., AWS,... ...(e.g., Kubernetes), and observability practices PREFERRED...CloudFull timeRemote work$167.7k - $245.2k
...securing workloads and AI infrastructure using advanced... ...Linux and Windows.Our software agent team builds high-... ...in on-premises and cloud environments. These... ...part of the Windows agent engineering team, you will focus on... ...Message to applicants applying to work in the U.S. and...CloudFull timeTemporary workLocal areaRemote workFlexible hours- ...India, save your time and DO NOT APPLY You will have to submit a... ...of sponsorship Senior Software Engineer (Full-Stack / Systems)... ...delivering custom platforms, AI systems, automation tools, and... ...applications ~ Strong knowledge of cloud infrastructure (AWS, GCP, or...CloudRemote work
$207.96k - $218.9k
...more about working at Coinbase. As a Senior Software Engineer on the USDC team within the Institutional... ...relational data stores, gRPC/Protobuf, and observability tooling in a cloud-native environment. Utilizes generative AI responsibly, maintaining human oversight to...CloudLocal area$186.07k - $218.9k
...working at Coinbase. As a Senior Software Engineer within the Security Platform... ...Key Infrastructure (PKI) and applied cryptography, with proven... ...private key material with HSMs or cloud KMS Extensive programming... ...Utilizes generative AI responsibly, maintaining human...CloudLocal area$132.23k - $176.31k
...the trusted network for the AI‑powered world, connecting people... ...connectivity across cloud, edge, and AI workloads for enterprises... ...close partnership with engineering, analytics, Adobe, Salesforce... ...modeling, ingestion, orchestration, observability, testing, security, and...CloudFull timeTemporary work- ...origination and servicing software for banks, credit... ...Senior Software Engineers to build and... ...Server data, and the cloud infrastructure... ...we are expanding AI-assisted development... ...before it ships. Apply the testing... ...CI/CD pipelines Observability OpenTelemetry (...Cloud
$121.4k - $218.6k
...design and deliver secure cloud solutions for... ...Our Custom Government Engineering Team! The Custom Government... ...highly productive Senior Software Engineer with a solid systems... ...Guiding code reviews, applying security patches, and... ...Agile practices and AI-assisted workflows Partnering...CloudPermanent employmentWork experience placementWork at officeWork from homeFlexible hours$99.6k - $192.9k
...cutting-edge, on-premises and cloud analytic solutions. We are... ...portfolio by embedding Generative AI, Large Language Models (LLMs)... ...industrial data. This Software Engineer position is ideal for a technically... ...Company, we encourage you to apply! As an established global...CloudImmediate startRemote workFree visaFlexible hours$110k - $165k
...future of our communities. This is a Lead Cloud & Infrastructure Engineering position at Vice President Level,... ..., security, automation, and AI adoption efforts, partnering with engineering... ...scripting, or orchestration.Experience applying AI or GenAI tools to infrastructure...CloudTemporary workLocal areaWeekend work- ...Description Summary: The Principal Customer Solutions Engineer a highly skilled... ...user groups. Applies own subject matter knowledge... ...Knowledge of major Cloud Provider... .... . What Rocket Software can offer you in USA... ...Software, we believe AI is a critical enabler...CloudFull timeLocal areaRemote workWorldwide
$186.07k - $218.9k
...working at Coinbase. As a Senior Software Engineer on the Data Platform team... ...data services spanning cloud data warehouses, data lakes,... ...end-to-end data security and observability services, including access controls... .... Utilizes generative AI responsibly, maintaining...CloudLocal area$186.07k - $218.9k
...working at Coinbase. As a Senior Software Engineer on the Blockchain Networks... ...maintain APIs, SLOs, and observability for the systems you build,... ....g., Bitcoin, Ethereum) and apply that knowledge to simplify cross... ...Utilizes generative AI responsibly, maintaining human...Local area$116.3k - $264.1k
...service issues. Guides team-level observability, capacity planning, and... ...Key Responsibilities Platform Software Development: Operate a... ...to life-saving care. And with AI embedded across our products... ...company leading the way in AI and cloud solutions that impact billions...CloudTemporary workImmediate startFlexible hours$103.71k - $138.28k
...Lumen is the trusted network for the AI‑powered world, connecting people, data, and... ...secure, high‑performance connectivity across cloud, edge, and AI workloads for enterprises,... ...join us today. The Role The Field Engineering Quality Control Engineer is responsible for...CloudFull timeTemporary workFor contractors$128.19k - $184.01k
...build with a wide variety of AI language models (bots), including... ...We are seeking a talented iOS Engineer to join us in building Poe, an... ...experience Own the entire software development process from... ...underrepresented groups in tech, to apply for our job openings. We...Remote jobFull time$152.41k - $179.3k
...customers. Support the team's engineering quality through code reviews,... ...and Experience: 2+ years of software engineering experience building... ...languages (e.g., Go, Java) and cloud infrastructure (e.g., AWS, GCP... .... Utilizes generative AI responsibly, maintaining human...CloudContract workLocal area$186.07k - $218.9k
...the Consumer & Business group as a Senior Software Engineer, Backend helping scale Coinbase Advanced... ..., Terraform, Datadog, or similar cloud infrastructure tools, and working knowledge... ...technical judgment. Utilizes generative AI responsibly, maintaining human oversight...CloudLocal areaWorldwide
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Principal Software Engineer, Applied AI - Splunk Observability Cloud. Be the first to apply!



