Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Senior Splunk & Observability Engineer

System One

Job Description

Job Description

Job Title: Senior Splunk & Observability Engineer Job Type: Permanent Full Time Location: Knoxville, TN, Columbia SC or Lafayette, LA or Birmingham, AL Position Description Seeking an experienced Senior Splunk & Observability Engineer to support enterprise scale observability platforms within a complex, AWS based production environment. This is a senior hands on role combining advanced Splunk administration and engineering, AWS production support, observability, and automation. The engineer will manage and support Splunk Enterprise running on AWS, working across the full data lifecycle from source onboarding and ingestion through indexing, search, dashboards, alerts, retention, and recovery. Responsibilities include developing and optimizing SPL queries, troubleshooting ingestion and performance issues, improving platform stability, supporting upgrades and vulnerability remediation, and restoring or replaying data following service interruptions. The role will also work extensively with AWS, Amazon CloudWatch, OpenTelemetry, and Python based automation to improve enterprise monitoring and resiliency. The engineer will build and maintain OpenTelemetry agents and collectors, support observability integrations across technology stacks, and help automate platform operations and vulnerability remediation. This position is required in one of the following locations: Lafayette, LA, Knoxville, TN, Columbia, SC, Birmingham, AL Your future duties and responsibilities . Administer, engineer, upgrade, and support Splunk Enterprise in AWS. . Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2. . Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery. . Develop and optimize complex SPL queries and troubleshoot query performance issues. . Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage. . Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues. . Restore data flow and replay data following ingestion or platform interruptions. . Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs. . Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts. . Develop Python based automation for observability and platform operations. . Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms. . Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation. . Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments. . Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams. . Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes. Required qualifications to be successful in this role . 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments. . Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery. . Advanced SPL query development and performance optimization skills. . Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS. . Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch. . Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases. . Strong understanding of OpenTelemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration. . Experience building or maintaining OpenTelemetry agent and collector packages. . Working knowledge of Python for automation and operational tooling. . Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON. . Strong production troubleshooting skills, including ingestion failures, data volume issues, platform outages, performance problems, and data recovery. . Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes. . Ability to lead technical troubleshooting calls, communicate findings clearly, and coordinate effectively across cloud, application, security, database, and production support teams. Educational Requirements: ?Bachelor's degree in Computer Science, Information Systems, or a related field.?? Ref: #404-IT Pittsburgh

Vacancy posted 24 days ago
Similar jobs that could be interesting for youBased on the Senior Splunk & Observability Engineer in Knoxville, TN vacancy
  •  ...to work on site in either of the following locations : Lafayette, LA - Knoxville, TN - Columbia, SC - Birmingham, AL Splunk Observability Engineer : Hands on role supporting large Splunk and observability platforms in an AWS production setting. Run, upgrade and... 
    Suggested
    For contractors
    H1b
    Immediate start

    System One

    Knoxville, TN
    3 days ago
  •  ...documentation of SOP and safety requirements. Coordinates with IF warehouse and IT to resolve system issues affecting inventory validity. Observes all associates periodically to ensure proper SOP procedures are being followed and maintained. Assists warehouse leadership in... 
    Senior
    Flexible hours
    Shift work

    Melaleuca:The Wellness Company

    Knoxville, TN
    3 days ago
  •  ...Construction Materials Testing (CMT) Senior Field Technician IECS Limited is offering...  ...ResponsibilitiesField Activities: Perform observation, testing, sampling, and inspection of...  ...precision in accordance with accepted engineering practices.Career Development: As your experience... 
    Senior
    For contractors

    Engineering Consulting Services

    Knoxville, TN
    3 days ago
  • $132.23k - $176.31k

     ...connectivity, join us today. The Role The Senior Lead Marketing Data Platform Architect...  ...role requires close partnership with engineering, analytics, Adobe, Salesforce, and...  ...modeling, ingestion, orchestration, observability, testing, security, and environment promotion... 
    Senior
    Full time
    Temporary work

    Lumen

    Knoxville, TN
    1 day ago
  •  ...Description GIG South / Generation Industrial & Government The Senior Electrical Field Technician candidate should have experience...  ...electricians to implement work or modifications made by the engineering team. This position requires extensive travel or extended time... 
    Senior
    Relocation

    Mesa Associates, Inc.

    Knoxville, TN
    24 days ago
  •  ...and implementing engaging activities for residents. You will reflect diverse interests and lead programs that enrich daily life for seniors in a healthcare setting. The ideal candidate has HS diploma or GED, experience creating and supervising recreational programs... 
    Senior

    Hamister Group

    Knoxville, TN
    2 days ago
  • $162.45k - $357.65k

     ...is currently seeking a Director, Global Senior Cloud Security Architect to join our Global...  ...leadership, guidance, and mentorship to engineers, analysts, and developers, ensuring...  ...publishes a calendar of holidays to be observed during the year depending on your job classification... 
    Senior
    Work experience placement
    H1b
    Local area

    KPMG

    Knoxville, TN
    1 day ago
  •  ...Senior Psychiatric Technician Full Time, 72 Hours Per Pay Period, Night Shift Parkwest Overview: Parkwest Medical Center,...  ...of patient activities on the milieu as well as any specialized observation needs. Monitors vital signs, collects urine specimens, and... 
    Senior
    Full time
    Shift work
    Night shift

    Covenant Health

    Knoxville, TN
    3 days ago
  • KinderCare Learning Companies in Knoxville, TN is seeking a Sr. Teacher to lead curriculum implementation, mentor teachers, and create a safe, nurturing environment where children can play and learn. You'll partner with families to support their children's education...
    Senior

    Kindercare

    Knoxville, TN
    2 days ago
  • $170k - $180k

     ...firm, and we are recruiting for an experienced Geotechnical Engineer based in Knoxville, TN.  This direct hire role will be aligned...  ...Masonry inspections Deep foundation testing and observation Special inspections in accordance with applicable building... 
    Senior
    Weekly pay
    Full time
    Contract work
    For contractors
    Casual work
    Work at office
    Relocation package

    Johnson Service Group

    Knoxville, TN
    12 days ago
  •  ...Seeking a Senior Instructional Designer with 7+ years of experience in corporate learning to design effective learning journeys and solutions. The ideal candidate will have strong expertise in adult learning, instructional design, needs assessment, and performance consulting... 
    Senior

    Veriipro

    Knoxville, TN
    2 days ago
  •  ...About the Role Our client, a well-established financial services group, is seeking a detail-oriented and experienced Senior Financial Accountant to join their fully remote finance team. This role is responsible for preparing accurate and timely financial statements... 
    Senior
    Remote work

    Placements24

    Knoxville, TN
    3 days ago
  •  ...Job Description Job Description Job Title: Senior Cloud Security Architect Location: Birmingham, AL, Columbia, SC, Knoxville...  ...will work with product owners, enterprise architecture, engineering, and security teams to protect sensitive data, strengthen identity... 
    Senior
    Full time

    System One

    Knoxville, TN
    24 days ago
  • $100k - $130k

     ...and the ability to guide clients through difficult circumstances. At Mandy Hancock Law PLLC , we are looking for a seasoned Senior Family Law Attorney who is ready to bring that experience to a growing practice where quality matters and professional judgment is... 
    Senior

    Mandy Hancock Law

    Knoxville, TN
    4 days ago
  • $30 - $54 per hour

    Job Description Job Description Overview WHY COOLSYS? At CoolSys, we offer more than just a job—we provide stability, growth, and industry-leading benefits for our employees. No wonder Newsweek named CoolSys one of America’s Greatest Workplaces! As a large company...
    Senior
    Work experience placement

    CoolSys

    Knoxville, TN
    11 days ago
  • $120k - $160k

     ...Senior Heavy Civil Estimator | East Tennessee area | $120K–$160K An established heavy civil contractor is looking for an experienced Senior Estimator to take a lead role in estimating complex civil construction projects. This position is ideal for an estimator... 
    Senior
    Full time
    For contractors
    For subcontractor

    Hays

    Knoxville, TN
    3 days ago
  •  ...Consor in Tennessee is seeking a Senior Project Manager - Water/Wastewater to lead planning, design and delivery of municipal water, wastewater, and related projects while growing the firm's client base. You will develop relationships with municipal utilities, pursue... 
    Senior

    Consor

    Knoxville, TN
    2 days ago
  •  ...The Senior Account Manager serves as the primary liaison between the company and its key clients, ensuring strong relationships, high customer satisfaction, and continued business growth. This role focuses on understanding client needs, aligning Managed IT services to... 
    Senior

    Industry Analysts , Inc.

    Knoxville, TN
    2 days ago
  • $200k - $250k

     ...Job Title: Senior Counsel Location: Onsite (Chevy Chase, MD) Salary: $200,000-$250,000 Skills: enclosed mall, shopping center, retail, commercial, leasing, leases, reciprocal easement agreements, REAs, development agreements, acquisitions, dispositions, real... 
    Senior

    Talently

    Knoxville, TN
    3 days ago
  • $77.33k - $157.67k

     ...future, and what we stand for as a firm.KPMG is currently seeking a Senior Associate to join our State and Local Tax (SALT) practice....  ...Additionally, each year KPMG publishes a calendar of holidays to be observed during the year depending on your job classification and entity... 
    Senior
    Local area
    Remote work

    KPMG

    Knoxville, TN
    4 days ago
  • Summit General Contractors, Inc. in Knoxville, TN, is seeking a Project Manager to oversee commercial construction projects from inception to completion. You will lead project teams, coordinate subcontractors and suppliers, and ensure projects meet scope, schedule, and...
    Senior
    For contractors
    For subcontractor

    Summit General Contractors, Inc.

    Knoxville, TN
    3 days ago
  •  ...Job Objective The Tax Senior CPA provides accurate tax preparation and advisory services while building strong client relationships. With 1 to 4 years of experience, this role focuses on developing technical skills, applying sound judgment, and supporting the firm'... 
    Senior
    Local area

    Addicks CPA Firm LLP

    Knoxville, TN
    3 days ago
  •  ...DevOps EngineerSeeking a DevOps Engineer to support secure, reliable, and scalable software delivery within a highly regulated financial services environment. This role focuses on CI/CD engineering, cloud infrastructure automation, Infrastructure as Code, and production... 
    Senior

    System One Holdings, LLC

    Knoxville, TN
    4 days ago
  •  ...Senior Roadway and Bridge InspectorRK&K is seeking talented Senior Roadway and Bridge Inspectors to work in the Knoxville/Tri-Cities area, with a focus in providing construction engineering inspection services on transportation projects for clients within the area. The... 
    Senior
    For contractors
    Local area

    RK&K

    Knoxville, TN
    3 days ago
  • $70k - $80k

     ...Growing Reputable Law Firm | 70-80K DOE + Incredible Benefits! Salary: $70,000 - $80,000 per year We are seeking an experienced Senior Accounts Payable Specialist to support a high-volume AP function within a fast-paced professional services environment. This... 
    Senior
    Local area
    Remote work

    Jobot

    Knoxville, TN
    1 day ago
  • $90.8k - $136.2k

     ...analysis; familiarity with SQL, MongoDB querying, Amplitude, Splunk, and/or Figma is helpful. ~ Systems analysis: ability to analyze...  ...and translating them into technical specifications engineering can build from. ~ Strong analytical and problem-solving skills... 
    Senior
    Full time
    Flexible hours
    Shift work

    World Wide Technology

    Knoxville, TN
    3 days ago
  •  ...who we are.Who You AreWe're looking for an energetic, driven sales associate ready to take their career to the next level. As the Senior Stylist you are the right-hand to the Store Leader, personal styling coach to the broader team, and community cheerleader. You are... 
    Senior
    Flexible hours
    Afternoon shift

    Vow'd Weddings

    Knoxville, TN
    22 hours ago
  •  ...Job Title - Senior API Platform Software Engineer Locations: Lafayette, LA | Knoxville, TN | Birmingham, AL | Columbia...  .... . Familiarity with monitoring and observability tools such as OpenTelemetry (OTEL), Splunk, and AWS CloudWatch. . Understanding of... 
    Senior
    Full time

    System One

    Knoxville, TN
    7 days ago
  • $77.2k - $96.5k

    Qualifications: ~ Bachelor’s degree in business, Accounting, MIS, Finance, or related field ~3+ years supporting or implementing Oracle EBS R12 and/or Oracle Cloud Financials ~ Hands-on experience with Order-to-Cash (O2C) and/or Procure-to-Pay (P2P) processes ~...
    Senior
    Full time
    Shift work

    World Wide Technology

    Knoxville, TN
    2 days ago
  • $89.6k - $123.2k

     ...Position Description: CGI is seeking a Senior Kubernetes Platform Engineer to build, manage, and scale production Amazon EKS environments in a fast...  ...and resilience testing, and establish robust observability practices across metrics, logs, traces, and alerting. Security... 
    Senior
    Full time
    Local area

    CGI

    Knoxville, TN
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Senior Splunk & Observability Engineer. Be the first to apply!