Cribl Engineer

United States

Suitability/Public Trust

Fully remote

Information Technology

Overview

GovCIO is currently hiring for Cribl Engineer to support our Administrative Office of the US Courts NLS project. The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote.

Responsibilities

• Develop apply best practices and tools for data ingestion, indexing, and management to optimize data sources and refine data collection processes to capture only pertinent data.

• Plan and perform Cribl platform upgrades (Leader, Worker, and Edge nodes) following defined change control procedures.

• Manage and optimize the Cribl distributed infrastructure, ensuring scalability, stability, and efficient data routing.

• Continuously monitor Cribl performance, including throughput, queue depth, and worker health metrics.

• Develop and maintain Cribl pipelines for new data sources, implementing filtering, sampling, and enrichment logic.

• Migrate existing Splunk forwarder-based data inputs to Cribl for improved control and flexibility.

• Build and maintain Cribl Packs for standardized configurations across multiple environments.

• Implement data reduction and enhancement workflows to minimize ingestion volume and improve data quality.

• Maintain and enhance Ansible playbooks for automated deployments, configurations, and upgrades.

• Integrate GitOps CI/CD pipelines (e.g., GitLab, Jenkins, Terraform) to manage configuration-as-code for both Splunk and Cribl.

• Develop, test, and review merge requests related to dashboards, alerts, saved searches, and data onboarding pipelines.

• Perform Splunk core upgrades (indexers, search heads, cluster masters, deployers) ensuring backward compatibility and minimal downtime.

• Upgrade and validate Splunk Add-ons and Apps, maintaining functionality and CIM compliance.

• Develop and maintain custom props, transforms, eventtypes, and lookups to normalize data consistently.

• Ensure CIM compliance for all add-ons and sourcetypes used across the platform.

• Handle escalations from Operations and perform deep-dive troubleshooting on ingestion, parsing, or performance issues.

• Perform break/fix analysis on Splunk core services such as KVStore, clustering, deployment server, and scheduler.

• Conduct performance tuning for search optimization, bucket management, and scheduler balancing across SHC.

• Design and maintain retention, archival, and index management strategies to align with business and compliance goals.

• Manage license allocation, volume forecasting, and capacity planning across indexer clusters.

• Develop and maintain monitoring and alerting integrations for Cribl and Splunk infrastructure health.

• Collaborate with Operations on incident triage, root cause analysis, and postmortem documentation.

• Create and maintain runbooks and engineering guides for deployments, upgrades, and troubleshooting.

• Participate in architecture and design discussions to ensure Splunk and Cribl meet enterprise scaling and reliability needs.

• Implement security and compliance controls including token rotation, TLS configurations, and secret management via Vault or GCP Secret Manager.

• Perform disaster recovery testing and validate replication and failover processes across clusters.

• Collaborate with governance teams to align on data retention, anonymization, and privacy requirements.

• Support continuous improvement by analyzing ingestion efficiency, performance benchmarks, and automation opportunities.

• Lead knowledge-sharing sessions and technical handoffs with Operations for newly deployed features or pipelines.

Qualifications

Bachelor's with 8+ years (or commensurate experience)

Required Skills and Experience

• 3 years of experience with Cribl data engine

• Understanding of Data from a Security Perspective

Clearance Required: Must be able to attain and maintain AOUSC Public Trust

Preferred Skills and Experience

• Cribl Admin Cert

• Experience in a Splunk Environment

• Cloud Experience

Posted Salary Range

USD $105,000.00 - USD $145,000.00 /Yr.

Company Overview

GovCIO is a team of transformers--people who are passionate about transforming government IT. Every day, we make a positive impact by delivering innovative IT services and solutions that improve how government agencies operate and serve our citizens.

But we can't do it alone. We need great people to help us do great things - for our customers, our culture, and our ability to attract other great people. We are changing the face of government IT and building a workforce that fuels this mission. Are you ready to be a transformer?

What You Can Expect

Interview & Hiring Process

If you are selected to move forward through the process, here’s what you can expect:

• During the Interview Process

• Virtual video interview conducted via video with the hiring manager and/or team

• Camera must be on

• A valid photo ID must be presented during each interview

• During the Hiring Process

• Enhanced Biometrics ID verification screening

• Background check, to include:

• Criminal history (past 7 years)

• Verification of your highest level of education

• Verification of your employment history (past 7 years), based on information provided in your application

Employee Perks

At GovCIO, we consistently hear that meaningful work and a collaborative team environment are two of the top reasons our employees enjoy working here. In addition, our employees have access to a range of perks and benefits to support their personal and professional well-being, beyond the standard company offered health benefits, including:

• Employee Assistance Program (EAP)

• Corporate Discounts

• Learning & Development platform, to include certification preparation content

• Training, Education and Certification Assistance*

• Referral Bonus Program

• Internal Mobility Program

• Pet Insurance

• Flexible Work Environment

*Available to full-time employees

Our employees’ unique talents and contributions are the driving force behind our success in supporting our customers, which ultimately fuels the success of our company. Join us and be a part of a culture that invests in its people and prioritizes continuous enhancement of the employee experience.

We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, disability, or status as a protected veteran. EOE, including disability/vets.

Posted Pay Range

The posted pay range, if referenced, reflects the range expected for this position at the commencement of employment, however, base pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, education, experience, and internal equity. The total compensation package for this position may also include other compensation elements, to be discussed during the hiring process. If hired, employee will be in an “at-will position” and the GovCIO reserves the right to modify base salary (as well as any other discretionary payment or compensation program) at any time, including for reasons related to individual performance, GovCIO or individual department/team performance, and market factors.