Cribl Engineer

Overview GovCIO is hiring a Cribl Engineer to support our Administrative Office of the US Courts NLS project. The NLS ingests an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote. Responsibilities Develop and apply best practices and tools for data ingestion, indexing, and management to optimize data sources and refine data collection processes to capture only pertinent data. Plan and perform Cribl platform upgrades (Leader, Worker, and Edge nodes) following defined change control procedures. Manage and optimize the Cribl distributed infrastructure, ensuring scalability, stability, and efficient data routing. Continuously monitor Cribl performance, including throughput, queue depth, and worker health metrics. Develop and maintain Cribl pipelines for new data sources, implementing filtering, sampling, and enrichment logic. Migrate existing Splunk forwarder-based data inputs to Cribl for improved control and flexibility. Build and maintain Cribl Packs for standardized configurations across multiple environments. Implement data reduction and enhancement workflows to minimize ingestion volume and improve data quality. Maintain and enhance Ansible playbooks for automated deployments, configurations, and upgrades. Integrate GitOps CI/CD pipelines (e.g., GitLab, Jenkins, Terraform) to manage configuration‑as‑code for both Splunk and Cribl. Develop, test, and review merge requests related to dashboards, alerts, saved searches, and data onboarding pipelines. Perform Splunk core upgrades (indexers, search heads, cluster masters, deployers) ensuring backward compatibility and minimal downtime. Upgrade and validate Splunk Add‑ons and Apps, maintaining functionality and CIM compliance. Develop and maintain custom props, transforms, eventtypes, and lookups to normalize data consistently. Ensure CIM compliance for all add‑ons and sourcetypes used across the platform. Handle escalations from Operations and perform deep‑dive troubleshooting on ingestion, parsing, or performance issues. Perform break/fix analysis on Splunk core services such as KVStore, clustering, deployment server, and scheduler. Conduct performance tuning for search optimization, bucket management, and scheduler balancing across SHC. Design and maintain retention, archival, and index management strategies to align with business and compliance goals. Manage license allocation, volume forecasting, and capacity planning across indexer clusters. Develop and maintain monitoring and alerting integrations for Cribl and Splunk infrastructure health. Collaborate with Operations on incident triage, root cause analysis, and postmortem documentation. Create and maintain runbooks and engineering guides for deployments, upgrades, and troubleshooting. Participate in architecture and design discussions to ensure Splunk and Cribl meet enterprise scaling and reliability needs. Implement security and compliance controls including token rotation, TLS configurations, and secret management via Vault or GCP Secret Manager. Perform disaster recovery testing and validate replication and failover processes across clusters. Collaborate with governance teams to align on data retention, anonymization, and privacy requirements. Support continuous improvement by analyzing ingestion efficiency, performance benchmarks, and automation opportunities. Lead knowledge‑sharing sessions and technical handoffs with Operations for newly deployed features or pipelines. Qualifications Bachelor's degree with 8+ years of relevant experience (or commensurate experience). Required Skills and Experience 3 years of experience with the Cribl data engine. Understanding of data from a security perspective. Preferred Skills and Experience Cribl Admin Certification. Experience in a Splunk environment. Cloud experience. Clearance Required Must be able to attain and maintain AOUSC Public Trust. Posted Salary Range USD $105,000.00 – $145,000.00 per year. We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, disability, or status as a protected veteran. EOE, including disability/vets. #J-18808-Ljbffr