Splunk Engineer

Splunk Engineer Hybrid

The candidate selected for this role will be part of the T. Rowe Price Reliability and Integrations Engineering team within the Technology Services Engineering group. The team supports observability and developer productivity platforms at T. Rowe Price. The Sr. Splunk Infrastructure Engineer will be responsible for supporting Splunk Enterprise, including managing Windows and Linux servers agents, automating infrastructure, configuration and day-to-day operations through Ansible, and performing troubleshooting, root cause analysis, and resolution of complex technical issues related to Splunk deployments.

Responsibilities:

• Support onboarding and maintenance of logs to Splunk from Windows, Linux and cloud-based sources
• Support platform upgrades including coordinating testing of upgrades with users of the platform
• Automating manual platform management processes through Ansible or other scripting tools/languages
• Troubleshooting incidents impacting the Splunk platform
• Evaluate the use and integration of third-party add-ons
• Coordinating and collaboration with users of the platform
• Develop training and documentation materials

Experience:

• Ability to troubleshoot and diagnose complex issues
• Able to demonstrate experience supporting technical users and conduct requirements analysis
• Can work independently with minimal guidance & oversight
• Experience with IT Service Management and familiarity with Incident & Problem management
• Highly skilled in identifying performance bottlenecks, identifying anomalous system behavior, and resolving root cause of service issues.
• Demonstrated ability to effectively work across teams and functions to influence design, operations, and deployment of highly available software
• Knowledge of standard methodologies related to security, performance, and disaster recovery

Required Technical Expertise:

• 3 years experience managing and configuring Splunk Enterprise and/or Splunk Cloud
• Experience with Splunk clustered deployment topology
• Experience with Linux and Windows agents for Splunk administration
• Experience in designing, developing, and deploying cloud-based solutions using AWS
• Experience in onboarding new data, configuration, creating new dashboards, extracting information through Splunk
• Experience with writing or modifying custom Splunk addons
• Demonstrated proficiency with scripting and automation (bash, python, other programming languages)
• Familiarity with Splunk rest APIs
• Strong scripting skills (e.g., Python, Bash) for automation and custom development.
• In-depth knowledge of log management, data onboarding, and SIEM principles.

Preferred Technical Experience:

• Splunk Certification (Admin or Architect)
• Experience with Ansible tower automations
• Experience using Gitlab
• Experience with large platform migration efforts
• Experience with AWS OpenSearch
• Experience with Cribl
• Expertise in language such as Java, Python. Implementation knowledge in data processing pipelines using programming languages like Java and Python to extract, transform, and load (ETL) data
• Create and maintain data models, ensuring efficient storage, retrieval, and analysis of large datasets
• Troubleshoot and resolve issues related to data processing, storage, and retrieval.
• 3-5 years Experience in designing, developing, and deploying data lakes using AWS native services (S3, Glue (Crawlers, ETL, Catalog), IAM, Terraform, Athena)
• Experience in development of systems for data extraction, ingestion and processing of large volumes of data
• Experience with data pipeline orchestration platforms
• Experience in Ansible/Terraform/Cloud Formation scripts and Infrastructure as Code scripting is required
• Implement version control and CI/CD practices for data engineering workflows to ensure reliable and efficient deployments
• Proficiency in implementing monitoring, logging, and alerting solutions for data infrastructure (e.g., Prometheus, Grafana)
• Proficiency in distributed Linux environments