Senior Splunk & Observability Engineer
Temporary
System One
Job Title: Senior Splunk & Observability Engineer
Job Type: Permanent Full Time
Location: Knoxville, TN, Columbia SC or Lafayette, LA or Birmingham, AL Position Description
Seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based production environment. This is a senior hands on role combining advanced Splunk administration and engineering, AWS production support, observability, and automation. The engineer will manage and support Splunk Enterprise running on AWS, working across the full data lifecycle from source onboarding and ingestion through indexing, search, dashboards, alerts, retention, and recovery. Responsibilities include developing and optimizing SPL queries, troubleshooting ingestion and performance issues, improving platform stability, supporting upgrades and vulnerability remediation, and restoring or replaying data following service interruptions. The role will also work extensively with AWS, Amazon CloudWatch, OpenTelemetry, and Python based automation to improve enterprise monitoring and resiliency. The engineer will build and maintain OpenTelemetry agents and collectors, support observability integrations across technology stacks, and help automate platform operations and vulnerability remediation. This position is required in one of the following locations: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL Your future duties and responsibilities
. Administer, engineer, upgrade, and support Splunk Enterprise in AWS.
. Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2.
. Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery.
. Develop and optimize complex SPL queries and troubleshoot query performance issues.
. Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage.
. Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues.
. Restore data flow and replay data following ingestion or platform interruptions.
. Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs.
. Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts.
. Develop Python based automation for observability and platform operations.
. Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms.
. Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation.
. Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments.
. Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams.
. Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes. Required qualifications to be successful in this role
. 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments.
. Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery.
. Advanced SPL query development and performance optimization skills.
. Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS.
. Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch.
. Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases.
. Strong understanding of OpenTelemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration.
. Experience building or maintaining OpenTelemetry agent and collector packages.
. Working knowledge of Python for automation and operational tooling.
. Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON.
. Strong production troubleshooting skills, including ingestion failures, data volume issues, platform outages, performance problems, and data recovery.
. Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes.
. Ability to lead technical troubleshooting calls, communicate findings clearly, and coordinate effectively across cloud, application, security, database, and production support teams. Educational Requirements:
?Bachelor's degree in Computer Science, Information Systems, or a related field.?? Ref: #404-IT Pittsburgh
Job Type: Permanent Full Time
Location: Knoxville, TN, Columbia SC or Lafayette, LA or Birmingham, AL Position Description
Seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based production environment. This is a senior hands on role combining advanced Splunk administration and engineering, AWS production support, observability, and automation. The engineer will manage and support Splunk Enterprise running on AWS, working across the full data lifecycle from source onboarding and ingestion through indexing, search, dashboards, alerts, retention, and recovery. Responsibilities include developing and optimizing SPL queries, troubleshooting ingestion and performance issues, improving platform stability, supporting upgrades and vulnerability remediation, and restoring or replaying data following service interruptions. The role will also work extensively with AWS, Amazon CloudWatch, OpenTelemetry, and Python based automation to improve enterprise monitoring and resiliency. The engineer will build and maintain OpenTelemetry agents and collectors, support observability integrations across technology stacks, and help automate platform operations and vulnerability remediation. This position is required in one of the following locations: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL Your future duties and responsibilities
. Administer, engineer, upgrade, and support Splunk Enterprise in AWS.
. Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2.
. Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery.
. Develop and optimize complex SPL queries and troubleshoot query performance issues.
. Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage.
. Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues.
. Restore data flow and replay data following ingestion or platform interruptions.
. Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs.
. Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts.
. Develop Python based automation for observability and platform operations.
. Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms.
. Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation.
. Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments.
. Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams.
. Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes. Required qualifications to be successful in this role
. 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments.
. Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery.
. Advanced SPL query development and performance optimization skills.
. Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS.
. Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch.
. Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases.
. Strong understanding of OpenTelemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration.
. Experience building or maintaining OpenTelemetry agent and collector packages.
. Working knowledge of Python for automation and operational tooling.
. Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON.
. Strong production troubleshooting skills, including ingestion failures, data volume issues, platform outages, performance problems, and data recovery.
. Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes.
. Ability to lead technical troubleshooting calls, communicate findings clearly, and coordinate effectively across cloud, application, security, database, and production support teams. Educational Requirements:
?Bachelor's degree in Computer Science, Information Systems, or a related field.?? Ref: #404-IT Pittsburgh
Vacancy posted 8 days ago
Similar jobs that could be interesting for youBased on the Senior Splunk & Observability Engineer in Knoxville, TN vacancy
- ...Lead ML Engineer, Document Intelligence Platform We’re hiring a hands-on technical leader to own our end-to-end document intelligence... ...OCR + VLM + rules + human-review system design ML/LLM observability using MLflow, Langfuse, W&B, or similar Priority...Senior
Medici Land Governance Inc.
Knoxville, TN4 days ago - ...certified Construction Materials Testing (CMT) Senior Field Technician I to join our... ...Field Activities : Perform observation, testing, sampling, and inspection of materials... ...precision in accordance with accepted engineering practices. Career Development...SeniorFor contractors
ECS Limited
Knoxville, TN2 days ago - ...Senior Full stack Engineer - AI-Native Product Team Location: Remote (EST/CST) I’m working with a remote-first technology company in... ...continuously improve engineering practices, reliability, observability, and developer experience Ship quickly, learn from users...SeniorRemote work
EAMES CONSULTING, LLC
Knoxville, TN4 days ago $170k - $180k
...firm, and we are recruiting for an experienced, Geotechnical Engineer based in Knoxville, TN. This direct hire role will be aligned... ...Masonry inspections Deep foundation testing and observation Special inspections in accordance with applicable building...SeniorFull timeFor contractorsCasual workWork at officeLocal areaRelocation packageJSG (Johnson Service Group, Inc.)
Knoxville, TN1 day ago$175k - $200k
...The role We're hiring a Senior Software Engineer to be the hands-on technical owner of our live platform. You'll take over a production TypeScript... ...going. ~ Comfort owning production: deployments, observability, debugging live issues and incident response. ~...SeniorFor contractorsRival Inc
Knoxville, TN3 days ago- ...Senior Product Designer Lead the design of thoughtful, scalable digital experiences that are rooted in user needs, aligned with business... ...prototypes, and high-fidelity UI Partner with product and engineering in triad model Participate in prioritization, roadmap...Senior
CBTS Inc
Knoxville, TN4 days ago - 120K-160K About the Role The Senior Estimator is responsible for leading the estimation process for complex construction projects, overseeing the development of accurate and timely cost estimates. This role requires extensive experience in construction estimating...Senior
Hays
Knoxville, TN3 days ago - ...Senior Healthcare Infrastructure, Cloud & Security Engineer Position Summary United Theranostics is seeking a highly experienced Senior Healthcare IT Infrastructure & Cloud Engineer to lead the design, implementation, security, and ongoing management of our enterprise...Senior
United Theranostics
Knoxville, TN3 days ago - ...in Knoxville, TN is seeking a Life Enrichment Director to plan, organize, coordinate and facilitate activities for residents in a senior living community. This role also leads and trains Life Enrichment support roles and ensures safe, high-quality service across community...Senior
StoryPoint
Knoxville, TN8 hours ago - ...Seeking a Senior Instructional Designer with 7+ years of experience in corporate learning to design effective learning journeys and solutions. The ideal candidate will have strong expertise in adult learning, instructional design, needs assessment, and performance consulting...SeniorContract work
Neshent Technologies
Knoxville, TNa month ago - Aerostar Environmental and Construction LLC is seeking an Environmental Project Manager in Knoxville to lead complex environmental projects, manage budgets, schedules, and staff, and coordinate subcontractors. The role requires strong client management and regulatory compliance...SeniorFor subcontractor
Aerostar Environmental and Construction LLC (AEC)
Knoxville, TN3 days ago - Consor in Tennessee seeks an experienced TPD Area Principal to lead transportation program delivery and provide technical solutions for infrastructure challenges. You will guide projects from proposal through execution, shaping strategy to win new work while strengthening...Senior
- ...We are seeking a Senior Environmental Manager for our Knoxville, TN office to assist with the management and leadership of personnel... ...REQUIREMENTS: • Bachelor’s degree in environmental science, engineering, geology, or closely related environmental field (higher level...SeniorFull timeWork at office
Fisher Arnold
Knoxville, TN21 hours ago - ...Ciresiandmorek is seeking a senior manufacturing leader to oversee complex operations in Knoxville, TN. The role focuses on delivering safe, reliable, and cost-effective performance while balancing daily execution with longer-term improvement initiatives. The successful...Senior
$65k
...driven go-getter with GRIT, TENACITY , and an unstoppable desire to WIN ? If so, we want to hear from you! Overview: The Senior Business Development Manager is responsible for selling the services of Circle Logistics. This role is to prospect future clients,...SeniorFull timeCircle Logistics, Inc
Knoxville, TN13 hours ago- New Business Account Manager — Social/Digital Media Advertising | Remote US Ready to own your book of business and build something from the ground up? A leading social media platform is looking for a hunter-mentality Account Executive to drive new business in the native...SeniorContract workRemote work
HireTalent
Knoxville, TN3 days ago - ...Senior Mechanical Estimator Build More with Your Career at Christman Are you interested... ..., and mentor Estimators and Estimating Engineers. Develops and oversees conceptual... ...subcontractors. Attention to detail: observant, accomplishes task accurately the first...SeniorFor contractorsFor subcontractorWork visa
- ...Are you an experienced Senior IT Systems Analyst with a desire to excel? If so, then Talent Software Services may have the job for you! Our client is seeking an experienced Senior IT Systems Analyst to work in Rochester, MN. We are seeking an experienced professional...Senior
Jobrapido
Knoxville, TN9 hours ago - ...Seize the Opportunity As a Senior Technician, you're not just fixing equipment—you're powering industry, mentoring talent, and driving excellence. Why WMH? Because your career deserves more! At WMH ( we don't just offer jobs—we build unstoppable careers. With over...Senior
Carolina Division
Knoxville, TN2 days ago - ...About the Role Smart Build LLC and Smart Coats Inc. are seeking an experienced part-time Senior Accountant to work with our internal finance team on the review and cleanup of both companies' QuickBooks Online records. Our internal team already handles day-to-...SeniorPart time
Smart Build Inc
Knoxville, TN3 days ago $100k - $140k
...About the Position The company is seeking an experienced Senior Estimator to join the preconstruction team in either Atlanta,... ...ranging from $5M–$50M Collaborate with project managers, engineers, and leadership during preconstruction Solicit and evaluate...SeniorFor contractorsFor subcontractor- ...The Tax Senior Associate is an advanced tax professional with a strong foundation in tax compliance and planning and emerging foundation in long-term tax and financial strategies that support client wealth and estate planning. The Tax Senior Associate role requires a...SeniorFull timeSeasonal workFlexible hours
The Mather Group, LLC
Knoxville, TN2 days ago - ...States. This is a good opportunity to join a growing and well-respected organization in the Nutraceutical sector. Position Title Senior Director of Sales Position Purpose and Objectives The Senior Director of Sales will drive new business growth and...SeniorRemote work
JBAndrews
Knoxville, TN3 days ago - SMB Senior Account Executive, Comcast Business Knoxville, Tennessee FreeWheelSenior
- ...financially sustainable programs that expand access to advanced wound care in their communities. About the Role We're looking for a Senior Account Executive with a proven track record in healthcare account management to manage and grow strategic relationships with our...SeniorFull timeContract workRemote work
Shared Health Services
Knoxville, TN21 hours ago - ...Who You Are We’re looking for an energetic, driven sales associate ready to take their career to the next level. As the Senior Stylist you are the right-hand to the Store Leader, personal styling coach to the broader team, and community cheerleader. You are...SeniorFull timeFlexible hoursWeekend workAfternoon shift
Vow'd Weddings
Knoxville, TN1 day ago $16.75 - $17 per hour
Assistant Store ManagerHourly rate ranges from $16.75 - $17.00 per hour and is dependent upon qualifications and experience.Benefits include: Company Paid Sick Time, Paid Vacation Time, Paid Holidays, Bereavement Pay, Jury Duty Pay, Contest Prize Awards, 401K Plan with...SeniorHourly payLocal areaSpencer Gifts
Knoxville, TN2 days ago- ...Geotechnical Senior Project Engineer ECS Limited is seeking an experienced Geotechnical Senior Project Engineer to manage key client projects in our Knoxville office. In this role, you will lead project execution, provide technical oversight, ensure quality control,...SeniorWork at office
Engineering Consulting Services
Knoxville, TN2 days ago - ...Senior Audit ManagerWe are seeking a highly skilled Senior Audit Manager to join our growing accounting firm. In this role, you will lead audit engagements, manage client relationships, and ensure the highest standards of audit quality and compliance. This position offers...Senior
gpac
Knoxville, TN2 days ago - ...cutting-edge AI, automation, and advanced strategy to deliver the highest ROI for our clients. Role Overview We’re seeking a Senior Media Buyer to own high-impact, performance-driven digital media programs for gaming clients operating in highly competitive, fast-...Senior
Acquire
Knoxville, TN3 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Splunk & Observability Engineer. Be the first to apply!
Related searches
- senior devops engineer remote Knoxville, TN
- senior information technology project manager Knoxville, TN
- senior vmware engineer Knoxville, TN
- sr electrical engineer Knoxville, TN
- senior cost engineer Knoxville, TN
- senior implementation project manager Knoxville, TN
- senior account manager Knoxville, TN
- senior robotics software engineer Knoxville, TN
- senior consulting engineer Knoxville, TN
- senior business manager Knoxville, TN