Senior Splunk & Observability Engineer
System One
Job Description
Job Description
Job Title: Senior Splunk & Observability Engineer Job Type: Permanent Full Time Location: Knoxville, TN, Columbia SC or Lafayette, LA or Birmingham, AL Position Description Seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based production environment. This is a senior hands on role combining advanced Splunk administration and engineering, AWS production support, observability, and automation. The engineer will manage and support Splunk Enterprise running on AWS, working across the full data lifecycle from source onboarding and ingestion through indexing, search, dashboards, alerts, retention, and recovery. Responsibilities include developing and optimizing SPL queries, troubleshooting ingestion and performance issues, improving platform stability, supporting upgrades and vulnerability remediation, and restoring or replaying data following service interruptions. The role will also work extensively with AWS, Amazon CloudWatch, OpenTelemetry, and Python based automation to improve enterprise monitoring and resiliency. The engineer will build and maintain OpenTelemetry agents and collectors, support observability integrations across technology stacks, and help automate platform operations and vulnerability remediation. This position is required in one of the following locations: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL Your future duties and responsibilities . Administer, engineer, upgrade, and support Splunk Enterprise in AWS. . Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2. . Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery. . Develop and optimize complex SPL queries and troubleshoot query performance issues. . Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage. . Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues. . Restore data flow and replay data following ingestion or platform interruptions. . Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs. . Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts. . Develop Python based automation for observability and platform operations. . Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms. . Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation. . Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments. . Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams. . Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes. Required qualifications to be successful in this role . 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments. . Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery. . Advanced SPL query development and performance optimization skills. . Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS. . Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch. . Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases. . Strong understanding of OpenTelemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration. . Experience building or maintaining OpenTelemetry agent and collector packages. . Working knowledge of Python for automation and operational tooling. . Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON. . Strong production troubleshooting skills, including ingestion failures, data volume issues, platform outages, performance problems, and data recovery. . Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes. . Ability to lead technical troubleshooting calls, communicate findings clearly, and coordinate effectively across cloud, application, security, database, and production support teams. Educational Requirements: ?Bachelor's degree in Computer Science, Information Systems, or a related field.?? Ref: #404-IT Pittsburgh
- ...to work on site in either of the following locations : Lafayette, LA - Knoxville, TN - Columbia, SC - Birmingham, AL Splunk Observability Engineer : Hands on role supporting large Splunk and observability platforms in an AWS production setting. Run, upgrade and...SuggestedFor contractorsH1bImmediate start
- ...documentation of SOP and safety requirements. Coordinates with IF warehouse and IT to resolve system issues affecting inventory validity. Observes all associates periodically to ensure proper SOP procedures are being followed and maintained. Assists warehouse leadership in...SeniorFlexible hoursShift work
- ...Construction Materials Testing (CMT) Senior Field Technician IECS Limited is offering... ...ResponsibilitiesField Activities: Perform observation, testing, sampling, and inspection of... ...precision in accordance with accepted engineering practices.Career Development: As your experience...SeniorFor contractors
$132.23k - $176.31k
...connectivity, join us today. The Role The Senior Lead Marketing Data Platform Architect... ...role requires close partnership with engineering, analytics, Adobe, Salesforce, and... ...modeling, ingestion, orchestration, observability, testing, security, and environment promotion...SeniorFull timeTemporary work- ...Description GIG South / Generation Industrial & Government The Senior Electrical Field Technician candidate should have experience... ...electricians to implement work or modifications made by the engineering team. This position requires extensive travel or extended time...SeniorRelocation
- ...and implementing engaging activities for residents. You will reflect diverse interests and lead programs that enrich daily life for seniors in a healthcare setting. The ideal candidate has HS diploma or GED, experience creating and supervising recreational programs...Senior
$162.45k - $357.65k
...is currently seeking a Director, Global Senior Cloud Security Architect to join our Global... ...leadership, guidance, and mentorship to engineers, analysts, and developers, ensuring... ...publishes a calendar of holidays to be observed during the year depending on your job classification...SeniorWork experience placementH1bLocal area- ...Senior Psychiatric Technician Full Time, 72 Hours Per Pay Period, Night Shift Parkwest Overview: Parkwest Medical Center,... ...of patient activities on the milieu as well as any specialized observation needs. Monitors vital signs, collects urine specimens, and...SeniorFull timeShift workNight shift
- KinderCare Learning Companies in Knoxville, TN is seeking a Sr. Teacher to lead curriculum implementation, mentor teachers, and create a safe, nurturing environment where children can play and learn. You'll partner with families to support their children's education...Senior
$170k - $180k
...firm, and we are recruiting for an experienced Geotechnical Engineer based in Knoxville, TN. This direct hire role will be aligned... ...Masonry inspections Deep foundation testing and observation Special inspections in accordance with applicable building...SeniorWeekly payFull timeContract workFor contractorsCasual workWork at officeRelocation package- ...Seeking a Senior Instructional Designer with 7+ years of experience in corporate learning to design effective learning journeys and solutions. The ideal candidate will have strong expertise in adult learning, instructional design, needs assessment, and performance consulting...Senior
- ...About the Role Our client, a well-established financial services group, is seeking a detail-oriented and experienced Senior Financial Accountant to join their fully remote finance team. This role is responsible for preparing accurate and timely financial statements...SeniorRemote work
- ...Job Description Job Description Job Title: Senior Cloud Security Architect Location: Birmingham, AL, Columbia, SC, Knoxville... ...will work with product owners, enterprise architecture, engineering, and security teams to protect sensitive data, strengthen identity...SeniorFull time
$100k - $130k
...and the ability to guide clients through difficult circumstances. At Mandy Hancock Law PLLC , we are looking for a seasoned Senior Family Law Attorney who is ready to bring that experience to a growing practice where quality matters and professional judgment is...Senior$30 - $54 per hour
Job Description Job Description Overview WHY COOLSYS? At CoolSys, we offer more than just a job—we provide stability, growth, and industry-leading benefits for our employees. No wonder Newsweek named CoolSys one of America’s Greatest Workplaces! As a large company...SeniorWork experience placement$120k - $160k
...Senior Heavy Civil Estimator | East Tennessee area | $120K–$160K An established heavy civil contractor is looking for an experienced Senior Estimator to take a lead role in estimating complex civil construction projects. This position is ideal for an estimator...SeniorFull timeFor contractorsFor subcontractor- ...Consor in Tennessee is seeking a Senior Project Manager - Water/Wastewater to lead planning, design and delivery of municipal water, wastewater, and related projects while growing the firm's client base. You will develop relationships with municipal utilities, pursue...Senior
- ...The Senior Account Manager serves as the primary liaison between the company and its key clients, ensuring strong relationships, high customer satisfaction, and continued business growth. This role focuses on understanding client needs, aligning Managed IT services to...Senior
$200k - $250k
...Job Title: Senior Counsel Location: Onsite (Chevy Chase, MD) Salary: $200,000-$250,000 Skills: enclosed mall, shopping center, retail, commercial, leasing, leases, reciprocal easement agreements, REAs, development agreements, acquisitions, dispositions, real...Senior$77.33k - $157.67k
...future, and what we stand for as a firm.KPMG is currently seeking a Senior Associate to join our State and Local Tax (SALT) practice.... ...Additionally, each year KPMG publishes a calendar of holidays to be observed during the year depending on your job classification and entity...SeniorLocal areaRemote work- Summit General Contractors, Inc. in Knoxville, TN, is seeking a Project Manager to oversee commercial construction projects from inception to completion. You will lead project teams, coordinate subcontractors and suppliers, and ensure projects meet scope, schedule, and...SeniorFor contractorsFor subcontractor
- ...Job Objective The Tax Senior CPA provides accurate tax preparation and advisory services while building strong client relationships. With 1 to 4 years of experience, this role focuses on developing technical skills, applying sound judgment, and supporting the firm'...SeniorLocal area
- ...DevOps EngineerSeeking a DevOps Engineer to support secure, reliable, and scalable software delivery within a highly regulated financial services environment. This role focuses on CI/CD engineering, cloud infrastructure automation, Infrastructure as Code, and production...Senior
- ...Senior Roadway and Bridge InspectorRK&K is seeking talented Senior Roadway and Bridge Inspectors to work in the Knoxville/Tri-Cities area, with a focus in providing construction engineering inspection services on transportation projects for clients within the area. The...SeniorFor contractorsLocal area
$70k - $80k
...Growing Reputable Law Firm | 70-80K DOE + Incredible Benefits! Salary: $70,000 - $80,000 per year We are seeking an experienced Senior Accounts Payable Specialist to support a high-volume AP function within a fast-paced professional services environment. This...SeniorLocal areaRemote work$90.8k - $136.2k
...analysis; familiarity with SQL, MongoDB querying, Amplitude, Splunk, and/or Figma is helpful. ~ Systems analysis: ability to analyze... ...and translating them into technical specifications engineering can build from. ~ Strong analytical and problem-solving skills...SeniorFull timeFlexible hoursShift work- ...who we are.Who You AreWe're looking for an energetic, driven sales associate ready to take their career to the next level. As the Senior Stylist you are the right-hand to the Store Leader, personal styling coach to the broader team, and community cheerleader. You are...SeniorFlexible hoursAfternoon shift
- ...Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia... .... . Familiarity with monitoring and observability tools such as OpenTelemetry (OTEL), Splunk, and AWS CloudWatch. . Understanding of...SeniorFull time
$77.2k - $96.5k
Qualifications: ~ Bachelor’s degree in business, Accounting, MIS, Finance, or related field ~3+ years supporting or implementing Oracle EBS R12 and/or Oracle Cloud Financials ~ Hands-on experience with Order-to-Cash (O2C) and/or Procure-to-Pay (P2P) processes ~...SeniorFull timeShift work$89.6k - $123.2k
...Position Description: CGI is seeking a Senior Kubernetes Platform Engineer to build, manage, and scale production Amazon EKS environments in a fast... ...and resilience testing, and establish robust observability practices across metrics, logs, traces, and alerting. Security...SeniorFull timeLocal area
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Senior Splunk & Observability Engineer. Be the first to apply!
- international tax senior Knoxville, TN
- senior sales Knoxville, TN
- senior network engineer remote Knoxville, TN
- senior manager legal Knoxville, TN
- senior account executive Knoxville, TN
- senior electrical designer Knoxville, TN
- senior windows systems engineer Knoxville, TN
- senior performance engineer Knoxville, TN
- remote senior salesforce administrator Knoxville, TN
- senior java j2ee developer Knoxville, TN







