Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Splunk & Observability Engineer Job Type: Permanent Full Time

System One Holdings, LLC

Senior Splunk & Observability Engineer Seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based production environment. This is a senior hands on role combining advanced Splunk administration and engineering, AWS production support, observability, and automation. The engineer will manage and support Splunk Enterprise running on AWS, working across the full data lifecycle from source onboarding and ingestion through indexing, search, dashboards, alerts, retention, and recovery. Responsibilities include developing and optimizing SPL queries, troubleshooting ingestion and performance issues, improving platform stability, supporting upgrades and vulnerability remediation, and restoring or replaying data following service interruptions. The role will also work extensively with AWS, Amazon CloudWatch, OpenTelemetry, and Python based automation to improve enterprise monitoring and resiliency. The engineer will build and maintain OpenTelemetry agents and collectors, support observability integrations across technology stacks, and help automate platform operations and vulnerability remediation. Your future duties and responsibilities include: Administer, engineer, upgrade, and support Splunk Enterprise in AWS. Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2. Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery. Develop and optimize complex SPL queries and troubleshoot query performance issues. Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage. Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues. Restore data flow and replay data following ingestion or platform interruptions. Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs. Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts. Develop Python based automation for observability and platform operations. Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms. Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation. Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments. Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams. Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes. Required qualifications to be successful in this role include: 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments. Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery. Advanced SPL query development and performance optimization skills. Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS. Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch. Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases. Strong understanding of OpenTelemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration. Experience building or maintaining OpenTelemetry agent and collector packages. Working knowledge of Python for automation and operational tooling. Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON. Strong production troubleshooting skills, including ingestion failures, data volume issues, platform outages, performance problems, and data recovery. Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes. Ability to lead technical troubleshooting calls, communicate findings clearly, and coordinate effectively across cloud, application, security, database, and production support teams. Educational Requirements: Bachelor's degree in Computer Science, Information Systems, or a related field. Ref: #404-IT Pittsburgh System One Holdings, LLC

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Senior Splunk & Observability Engineer Job Type: Permanent Full Time in Birmingham, AL vacancy
  • $175k - $225k

     ...The Benchmark Company, LLC—a full-service investment banking...  ...products and services to all types of investors.Whether you want...  ...this role you will need:Senior Sell Side Analyst with 15+ names...  ...on the position, is offered.Permanent, full-time, #LI-SD1 Job SummaryCategory... 
    Permanent employment
    Full time
    Senior
    Work at office

    StoneX

    Birmingham, AL
    4 days ago
  • Senior Technology Project Manager / Scrum Master Location: Lafayette, LA, Knoxville, TN, Birmingham, AL, Columbia, SC Type: Permanent Full Time Responsibilities Lead a defined workstream within a larger...  ..., especially regarding data engineering, SQL, data pipelines, and... 
    Permanent employment
    Full time
    Senior

    System One Holdings, LLC

    Birmingham, AL
    2 days ago
  • Job Title: Senior Atlassian Tools Administrator Location: Lafayette, LA, Birmingham...  ...AL, Columbia, SC or Knoxville, TN Job Type: Permanent Full Time Seeking a Senior Atlassian Tools Administrator...  ...Atlassian Cloud platform. The engineer will develop and maintain Atlassian... 
    Permanent employment
    Full time
    Senior

    System One Holdings, LLC

    Birmingham, AL
    2 days ago
  •  ...Job Title: Senior Vulnerability Management Analyst Location: Lafayette, LA, Knoxville...  ..., TN, Columbia, SC, Birmingham, AL Type: Permanent Fulltime Overview Seeking a...  ...s degree in Computer Science, Computer Engineering, Cybersecurity, Information Technology,... 
    Permanent employment
    Full time
    Senior
    Local area

    System One

    Birmingham, AL
    a month ago
  • Job Title: Senior ServiceNow Custom Application Developer Location: Lafayette, LA...  ...TN, Columbia, SC or Birmingham, AL Type: Permanent Full Time Responsibilities • Build and support...  ...Science, Information Technology, Engineering, or a related field, or equivalent professional... 
    Permanent employment
    Full time
    Senior
    Local area

    System One

    Birmingham, AL
    2 days ago
  •  ...Job Title: Senior Product Analyst/System Analyst Location: Birmingham, AL, Columbia, SC, Knoxville, TN, Lafayette, LA or Plano, TX. Job Type: Permanent Fulltime Responsibilities Analyze complex business processes, system interactions, and data flows within... 
    Permanent employment
    Full time
    Senior
    Local area

    System One

    Birmingham, AL
    a month ago
  • Senior CyberArk PAM Engineer Job Type: Permanent Full-Time Job Category: Cyber Security Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL Position Overview We are seeking a Senior CyberArk Privileged Access Management (PAM) Engineer to join an Identity & Access... 
    Permanent employment
    Full time
    Senior
    Local area

    System One

    Birmingham, AL
    2 days ago
  •  ...Senior Tax Associate - Birmingham, ALWe are seeking an experienced tax professional with...  ...high-quality client service.This is a full-time position available for immediate hire.Based...  ...consulting engagements including permanent establishment and nexus studies.Assist with... 
    Permanent employment
    Full time
    Senior
    Work at office
    Immediate start
    Flexible hours

    Staff Financial Group

    Birmingham, AL
    5 days ago
  • Senior Project Manager Birmingham or surrounding areas Permanent (Mixture of local office, and client site)Full time - with flexible working and core hoursAbout this opportunityGleeds is a leading global property and construction consultancy. Independent since 1875, we... 
    Permanent employment
    Full time
    Senior
    Contract work
    Work at office
    Local area
    Worldwide
    Flexible hours

    Gleeds

    Birmingham, AL
    2 days ago
  •  ...Job Description Job Description Senior Information Protection & AI Security Engineer Job Type: Permanent Full-Time Job Category: Cybersecurity Locations: Lafayette, LA | Knoxville, TN | Columbia, SC | Birmingham, AL Position Overview We are seeking a... 
    Permanent employment
    Full time
    Senior

    System One

    Birmingham, AL
    a month ago
  •  ...Job Description Job Description Senior AWS Python Developer * - Job Title: Senior AWS Python Developer * Job Category: Software Development/ Engineering Job Type: Permanent Full Time Location: Various, , United States No.Openings: 1 Description: Senior... 
    Permanent employment
    Full time
    Senior

    System One

    Birmingham, AL
    29 days ago
  •  ...Location Birmingham, AL | Outpatient Schedule / Job Type Contract | Full-Time | Days | 5 x 8-hour shifts | Expected 40 hours/week |...  ...license required (compact license accepted at start, but permanent Alabama state license must be obtained promptly) Prior... 
    Permanent employment
    Full time
    Weekly pay
    Contract work
    Immediate start
    Shift work

    Concentric Healthcare Staffing

    Birmingham, AL
    1 day ago
  • $155k - $185k

    Overview Permanent, full-timeConnecting clients to markets...  ...and services to all types of investors. Whether...  ...highly accomplished Senior Technical Software...  ...architecture, innovation, and engineering excellence of our...  ...testing strategies, observability, security,... 
    Permanent employment
    Senior
    Work at office

    StoneX

    Birmingham, AL
    7 hours ago
  •  ...Location Birmingham, AL | Outpatient Schedule / Job Type Contract | Full-Time | Days | 5×8-hour shifts | Expected 40 hours/week Key...  ...license required (compact license accepted at start, but permanent state licensure must be obtained promptly) Experience... 
    Permanent employment
    Full time
    Weekly pay
    Contract work
    Relief
    Immediate start
    Shift work
    Day shift

    Concentric Healthcare Staffing

    Birmingham, AL
    1 day ago
  •  ...50.00 Pay Band: SV-F Appointment Type: Fulltime Transportation-security operations...  ...procedures, operational information, observation, and professional judgment Review...  ...Supervisory TSO vacancies are limited to current permanent or eligible temporary TSA employees.... 
    Permanent employment
    Full time
    Temporary work
    Immediate start
    Trial period
    Flexible hours
    Shift work
    Night shift
    Weekend work
    Day shift
    Afternoon shift
    Early shift

    Airport Security Hiring

    Vestavia Hills, AL
    18 days ago
  •  ...50.00 Pay Band: SV-F Appointment Type: Fulltime Transportation-security operations...  ...procedures, operational information, observation, and professional judgment Review...  ...Supervisory TSO vacancies are limited to current permanent or eligible temporary TSA employees.... 
    Permanent employment
    Full time
    Temporary work
    Immediate start
    Trial period
    Flexible hours
    Shift work
    Night shift
    Weekend work
    Day shift
    Afternoon shift
    Early shift

    Federal Screening Careers

    Homewood, AL
    18 days ago
  •  ...Job Title: Senior AWS Java Full Stack Developer Location: Lafayette, LA / Knoxville, TN / Columbia, SC / Birmingham, AL / Plano, TX. Type: Permanent Full Time Work Model: Hybrid – onsite and remote We are hiring a Senior AWS Java Full... 
    Permanent employment
    Full time
    Senior
    Local area
    Remote work

    System One

    Birmingham, AL
    15 days ago
  • Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN...  ...Arrangement: Hybrid Employment Type: Full time Experience Required: 5+...  ...with monitoring and observability tools such as OpenTelemetry (OTEL), Splunk, and AWS CloudWatch. . Understanding... 
    Full time
    Senior

    System One Holdings, LLC

    Vestavia Hills, AL
    2 days ago
  •  ...eligible to obtain a U.S. Department of Defense security clearance. With few exceptions, this is restricted to U.S. citizens and legal permanent residents (a.k.a. current Green Card Holders). Tocaro Blue LLC is not able to sponsor work visas nor permanent resident cards ("... 
    Permanent employment
    Full time
    Senior
    Visa sponsorship
    Work visa

    Tocaro Blue

    Birmingham, AL
    more than 2 months ago
  •  ...Job Description Job Description Job Title: Senior AI Coding Engineer Duration: Full Time / Permanent Position Location: Birmingham, AL, Knoxville, TN or Lafayette, LA Work Mode: 5 Days Onsite Systemone is looking for a Senior AI Coding Engineer to lead... 
    Permanent employment
    Full time
    Senior

    System One

    Birmingham, AL
    11 days ago
  • Production Support Engineer - AWS Duration: Full Time / Permanent Role Location: Birmingham, AL, Columbia, SC, Knoxville, TN or Lafayette, LA Work Mode: 5 Days Onsite Position Overview Provide production support for critical HR, Finance, and other back-office applications... 
    Permanent employment
    Full time
    Senior
    Monday to Friday
    Flexible hours
    Night shift
    Rotating shift
    Weekend work

    System One Holdings, LLC

    Birmingham, AL
    2 days ago
  • $34 - $47 per hour

     ...qualified candidates Strong potential for long-term growth and permanent placement Requirements: Active RN license or...  ...proficiency (if education completed in a non-English-speaking institution) Job Type: Full-time Work Location: In person... 
    Permanent employment
    Full time
    Relocation package

    HR Journals LLC

    Homewood, AL
    2 days ago
  •  ...Senior DevSecOps Engineer Permanent Full-Time Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia SC Position Description This role focuses on improving software supply chain security, artifact management, open-source governance, CI... 
    Permanent employment
    Full time
    Senior

    System One

    Birmingham, AL
    a month ago
  •  ...Job Title : Senior Information Protection & AI Security Engineer Location: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL Type: Permanent Fulltime Overview Seeking a Senior Information Protection & AI Security Engineer to support... 
    Permanent employment
    Full time
    Senior

    System One

    Birmingham, AL
    a month ago
  •  ...Location Birmingham, AL | Outpatient Schedule / Job Type Contract | Full-Time | Days | Onsite | Expected 40 hours/week (32 hours/week...  ...Physical Therapist license (compact license accepted at start; permanent Alabama state license must be obtained promptly)... 
    Permanent employment
    Full time
    Contract work
    Relief
    Day shift

    IdealHire

    Birmingham, AL
    3 days ago
  •  ...Job Description Job Description Job Title: Senior GenAI & Data Engineer Duration: Full Time / Permanent Position Location: Birmingham, AL, Knoxville, TN, Columbia, SC or Lafayette, LA. Work Mode: Hybrid Systemone is seeking a Senior Gen AI Software Engineer... 
    Permanent employment
    Full time
    Senior

    System One

    Birmingham, AL
    21 days ago
  •  ...Job Description Job Description Job Title: Senior Ping Directory Engineer Duration: Full Time / Permanent Position Location: Birmingham, AL, Knoxville, TN, Columbia, SC or Lafayette, LA. Work Mode: 5 Days Onsite Systemone is seeking a Senior PingDirectory... 
    Permanent employment
    Full time
    Senior

    System One

    Birmingham, AL
    11 days ago
  • $140k

     ...Overview We're hiring Senior CNC Programmers for a precision manufacturing...  ...You'll work alongside manufacturing engineers and senior machinists to develop stable...  ...we're willing to pay. Let's chat. Full-time, permanent position Hiring on First Shift... 
    Permanent employment
    Full time
    Senior
    Day shift

    Laborup

    Birmingham, AL
    3 days ago
  •  ...Senior Project Manager – Healthcare Location: Birmingham, AL Reports to: Director of Project Management Employment Type: Full-time, salaried Compensation: Competitive salary + performance-based bonus + benefits About Creature Creature is a vertically... 
    Full time
    Senior
    Contract work

    Creature

    Birmingham, AL
    15 hours ago
  • $65k - $70k

     ...Senior AccountantOur client has an immediate need for a Senior Accountant in their Birmingham office. This is a permanent opportunity with excellent benefits. Who: Strong stable organization What: Senior Accountant When: Immediate need Where: Birmingham, AL Why: Newly... 
    Permanent employment
    Senior
    Work at office
    Immediate start

    Staff Financial Group

    Birmingham, AL
    4 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Splunk & Observability Engineer Job Type: Permanent Full Time. Be the first to apply!