Senior Splunk & Observability Engineer Job Type: Permanent Full Time
System One Holdings, LLC
Senior Splunk & Observability Engineer Seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based production environment. This is a senior hands on role combining advanced Splunk administration and engineering, AWS production support, observability, and automation. The engineer will manage and support Splunk Enterprise running on AWS, working across the full data lifecycle from source onboarding and ingestion through indexing, search, dashboards, alerts, retention, and recovery. Responsibilities include developing and optimizing SPL queries, troubleshooting ingestion and performance issues, improving platform stability, supporting upgrades and vulnerability remediation, and restoring or replaying data following service interruptions. The role will also work extensively with AWS, Amazon CloudWatch, OpenTelemetry, and Python based automation to improve enterprise monitoring and resiliency. The engineer will build and maintain OpenTelemetry agents and collectors, support observability integrations across technology stacks, and help automate platform operations and vulnerability remediation. Your future duties and responsibilities include: Administer, engineer, upgrade, and support Splunk Enterprise in AWS. Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2. Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery. Develop and optimize complex SPL queries and troubleshoot query performance issues. Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage. Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues. Restore data flow and replay data following ingestion or platform interruptions. Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs. Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts. Develop Python based automation for observability and platform operations. Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms. Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation. Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments. Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams. Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes. Required qualifications to be successful in this role include: 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments. Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery. Advanced SPL query development and performance optimization skills. Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS. Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch. Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases. Strong understanding of OpenTelemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration. Experience building or maintaining OpenTelemetry agent and collector packages. Working knowledge of Python for automation and operational tooling. Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON. Strong production troubleshooting skills, including ingestion failures, data volume issues, platform outages, performance problems, and data recovery. Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes. Ability to lead technical troubleshooting calls, communicate findings clearly, and coordinate effectively across cloud, application, security, database, and production support teams. Educational Requirements: Bachelor's degree in Computer Science, Information Systems, or a related field. Ref: #404-IT Pittsburgh System One Holdings, LLC
$175k - $225k
...The Benchmark Company, LLC—a full-service investment banking... ...products and services to all types of investors.Whether you want... ...this role you will need:Senior Sell Side Analyst with 15+ names... ...on the position, is offered.Permanent, full-time, #LI-SD1 Job SummaryCategory...Permanent employmentFull timeSeniorWork at office- Senior Technology Project Manager / Scrum Master Location: Lafayette, LA, Knoxville, TN, Birmingham, AL, Columbia, SC Type: Permanent Full Time Responsibilities Lead a defined workstream within a larger... ..., especially regarding data engineering, SQL, data pipelines, and...Permanent employmentFull timeSenior
- Job Title: Senior Atlassian Tools Administrator Location: Lafayette, LA, Birmingham... ...AL, Columbia, SC or Knoxville, TN Job Type: Permanent Full Time Seeking a Senior Atlassian Tools Administrator... ...Atlassian Cloud platform. The engineer will develop and maintain Atlassian...Permanent employmentFull timeSenior
- ...Job Title: Senior Vulnerability Management Analyst Location: Lafayette, LA, Knoxville... ..., TN, Columbia, SC, Birmingham, AL Type: Permanent Fulltime Overview Seeking a... ...s degree in Computer Science, Computer Engineering, Cybersecurity, Information Technology,...Permanent employmentFull timeSeniorLocal area
- Job Title: Senior ServiceNow Custom Application Developer Location: Lafayette, LA... ...TN, Columbia, SC or Birmingham, AL Type: Permanent Full Time Responsibilities • Build and support... ...Science, Information Technology, Engineering, or a related field, or equivalent professional...Permanent employmentFull timeSeniorLocal area
- ...Job Title: Senior Product Analyst/System Analyst Location: Birmingham, AL, Columbia, SC, Knoxville, TN, Lafayette, LA or Plano, TX. Job Type: Permanent Fulltime Responsibilities Analyze complex business processes, system interactions, and data flows within...Permanent employmentFull timeSeniorLocal area
- Senior CyberArk PAM Engineer Job Type: Permanent Full-Time Job Category: Cyber Security Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL Position Overview We are seeking a Senior CyberArk Privileged Access Management (PAM) Engineer to join an Identity & Access...Permanent employmentFull timeSeniorLocal area
- ...Senior Tax Associate - Birmingham, ALWe are seeking an experienced tax professional with... ...high-quality client service.This is a full-time position available for immediate hire.Based... ...consulting engagements including permanent establishment and nexus studies.Assist with...Permanent employmentFull timeSeniorWork at officeImmediate startFlexible hours
- Senior Project Manager Birmingham or surrounding areas Permanent (Mixture of local office, and client site)Full time - with flexible working and core hoursAbout this opportunityGleeds is a leading global property and construction consultancy. Independent since 1875, we...Permanent employmentFull timeSeniorContract workWork at officeLocal areaWorldwideFlexible hours
- ...Job Description Job Description Senior Information Protection & AI Security Engineer Job Type: Permanent Full-Time Job Category: Cybersecurity Locations: Lafayette, LA | Knoxville, TN | Columbia, SC | Birmingham, AL Position Overview We are seeking a...Permanent employmentFull timeSenior
- ...Job Description Job Description Senior AWS Python Developer * - Job Title: Senior AWS Python Developer * Job Category: Software Development/ Engineering Job Type: Permanent Full Time Location: Various, , United States No.Openings: 1 Description: Senior...Permanent employmentFull timeSenior
- ...Location Birmingham, AL | Outpatient Schedule / Job Type Contract | Full-Time | Days | 5 x 8-hour shifts | Expected 40 hours/week |... ...license required (compact license accepted at start, but permanent Alabama state license must be obtained promptly) Prior...Permanent employmentFull timeWeekly payContract workImmediate startShift work
$155k - $185k
Overview Permanent, full-timeConnecting clients to markets... ...and services to all types of investors. Whether... ...highly accomplished Senior Technical Software... ...architecture, innovation, and engineering excellence of our... ...testing strategies, observability, security,...Permanent employmentSeniorWork at office- ...Location Birmingham, AL | Outpatient Schedule / Job Type Contract | Full-Time | Days | 5×8-hour shifts | Expected 40 hours/week Key... ...license required (compact license accepted at start, but permanent state licensure must be obtained promptly) Experience...Permanent employmentFull timeWeekly payContract workReliefImmediate startShift workDay shift
- ...50.00 Pay Band: SV-F Appointment Type: Fulltime Transportation-security operations... ...procedures, operational information, observation, and professional judgment Review... ...Supervisory TSO vacancies are limited to current permanent or eligible temporary TSA employees....Permanent employmentFull timeTemporary workImmediate startTrial periodFlexible hoursShift workNight shiftWeekend workDay shiftAfternoon shiftEarly shift
- ...50.00 Pay Band: SV-F Appointment Type: Fulltime Transportation-security operations... ...procedures, operational information, observation, and professional judgment Review... ...Supervisory TSO vacancies are limited to current permanent or eligible temporary TSA employees....Permanent employmentFull timeTemporary workImmediate startTrial periodFlexible hoursShift workNight shiftWeekend workDay shiftAfternoon shiftEarly shift
- ...Job Title: Senior AWS Java Full Stack Developer Location: Lafayette, LA / Knoxville, TN / Columbia, SC / Birmingham, AL / Plano, TX. Type: Permanent Full Time Work Model: Hybrid – onsite and remote We are hiring a Senior AWS Java Full...Permanent employmentFull timeSeniorLocal areaRemote work
- Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN... ...Arrangement: Hybrid Employment Type: Full time Experience Required: 5+... ...with monitoring and observability tools such as OpenTelemetry (OTEL), Splunk, and AWS CloudWatch. . Understanding...Full timeSenior
- ...eligible to obtain a U.S. Department of Defense security clearance. With few exceptions, this is restricted to U.S. citizens and legal permanent residents (a.k.a. current Green Card Holders). Tocaro Blue LLC is not able to sponsor work visas nor permanent resident cards ("...Permanent employmentFull timeSeniorVisa sponsorshipWork visa
- ...Job Description Job Description Job Title: Senior AI Coding Engineer Duration: Full Time / Permanent Position Location: Birmingham, AL, Knoxville, TN or Lafayette, LA Work Mode: 5 Days Onsite Systemone is looking for a Senior AI Coding Engineer to lead...Permanent employmentFull timeSenior
- Production Support Engineer - AWS Duration: Full Time / Permanent Role Location: Birmingham, AL, Columbia, SC, Knoxville, TN or Lafayette, LA Work Mode: 5 Days Onsite Position Overview Provide production support for critical HR, Finance, and other back-office applications...Permanent employmentFull timeSeniorMonday to FridayFlexible hoursNight shiftRotating shiftWeekend work
$34 - $47 per hour
...qualified candidates Strong potential for long-term growth and permanent placement Requirements: Active RN license or... ...proficiency (if education completed in a non-English-speaking institution) Job Type: Full-time Work Location: In person...Permanent employmentFull timeRelocation package- ...Senior DevSecOps Engineer Permanent Full-Time Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia SC Position Description This role focuses on improving software supply chain security, artifact management, open-source governance, CI...Permanent employmentFull timeSenior
- ...Job Title : Senior Information Protection & AI Security Engineer Location: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL Type: Permanent Fulltime Overview Seeking a Senior Information Protection & AI Security Engineer to support...Permanent employmentFull timeSenior
- ...Location Birmingham, AL | Outpatient Schedule / Job Type Contract | Full-Time | Days | Onsite | Expected 40 hours/week (32 hours/week... ...Physical Therapist license (compact license accepted at start; permanent Alabama state license must be obtained promptly)...Permanent employmentFull timeContract workReliefDay shift
- ...Job Description Job Description Job Title: Senior GenAI & Data Engineer Duration: Full Time / Permanent Position Location: Birmingham, AL, Knoxville, TN, Columbia, SC or Lafayette, LA. Work Mode: Hybrid Systemone is seeking a Senior Gen AI Software Engineer...Permanent employmentFull timeSenior
- ...Job Description Job Description Job Title: Senior Ping Directory Engineer Duration: Full Time / Permanent Position Location: Birmingham, AL, Knoxville, TN, Columbia, SC or Lafayette, LA. Work Mode: 5 Days Onsite Systemone is seeking a Senior PingDirectory...Permanent employmentFull timeSenior
$140k
...Overview We're hiring Senior CNC Programmers for a precision manufacturing... ...You'll work alongside manufacturing engineers and senior machinists to develop stable... ...we're willing to pay. Let's chat. Full-time, permanent position Hiring on First Shift...Permanent employmentFull timeSeniorDay shift- ...Senior Project Manager – Healthcare Location: Birmingham, AL Reports to: Director of Project Management Employment Type: Full-time, salaried Compensation: Competitive salary + performance-based bonus + benefits About Creature Creature is a vertically...Full timeSeniorContract work
$65k - $70k
...Senior AccountantOur client has an immediate need for a Senior Accountant in their Birmingham office. This is a permanent opportunity with excellent benefits. Who: Strong stable organization What: Senior Accountant When: Immediate need Where: Birmingham, AL Why: Newly...Permanent employmentSeniorWork at officeImmediate start
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Splunk & Observability Engineer Job Type: Permanent Full Time. Be the first to apply!
- senior living Birmingham, AL
- senior accountant manufacturing Birmingham, AL
- senior manager customer operations Birmingham, AL
- senior full stack web developer Birmingham, AL
- senior magento developer Birmingham, AL
- senior sas administrator Birmingham, AL
- senior application support engineer Birmingham, AL
- senior software design engineer Birmingham, AL
- senior devops cloud engineer Birmingham, AL
- senior international tax analyst Birmingham, AL





