Principal Security Infrastructure Architect

Role: Principal Security Infrastructure Architect

Job Description:
Senior NDR & Platform Observability Engineer will support the operational health, visibility, and performance of the enterprise Network Detection & Response (NDR) environment, with a primary focus on the Corelight platform and surrounding telemetry pipelines. This role combines security operations expertise with the ability to build a modern monitoring and observability framework leveraging APIs, time series databases, automation, and data visualization tools.
The engineer will design and implement a comprehensive health monitoring architecture that ensures accurate, timely detection of platform degradation, enhanced visibility into sensor and pipeline performance, and operational insights that support Security Operations, Incident Response, and Network Engineering teams.

This role is responsible for:

• Operating and maintaining the NDR ecosystem.
• Developing automated collection of health and performance metrics using Python and REST APIs.
• Building a production ready observability stack using Grafana, Prometheus, InfluxDB, and Telegraf.
• Ensuring platform reliability, data quality, and visibility through dashboards, alerts, and automation workflows.
• Providing advanced troubleshooting support to ensure uninterrupted NDR coverage across the enterprise.
• The individual will play a critical role in improving detection efficacy, reducing noise, optimizing sensor uptime, and delivering insights that enhance the organization's overall security posture.

Key Responsibilities:

NDR Operations:

• Oversee daily operations of NDR sensors, appliances, and Zeek based detection pipelines.
• Monitor sensor health, data ingestion, packet throughput, and drop rates.
• Perform triage of NDR alerts and work with SOC/IR teams on escalations.
• Support tuning of Zeek scripts, Suricata rules, and Corelight detection packs.
• Identify data gaps, ingest delays, or coverage issues and drive resolution.
• Troubleshoot packet broker connections, SPAN/TAP feeds, and network visibility paths.
• Observability & Monitoring Architecture
• Design an enterprise grade observability solution for NDR platform and related telemetry systems.
• Build metrics collectors using Python to ingest REST API data into monitoring platforms.
• Integrate metrics into Prometheus, InfluxDB, or similar time series databases.
• Configure Telegraf pipelines for data collection, parsing, tagging, and forwarding.
• Develop dashboards and visualizations in Grafana for real time and historical performance analysis.
• Establish SLIs/SLOs related to NDR reliability, sensor uptime, ingest freshness, and data pipeline availability.

Automation & API Integration

• Develop Python automation scripts to standardize health checks, data validation, and system reporting.
• Integrate with SIEM, and packet broker APIs to extract key operational metrics.
• Build custom Prometheus exporters or collectors when native solutions are not available.
• Automate repetitive tasks such as sensor status checks, alert validation, and data integrity verification.

Documentation & Knowledge Transfer

• Create and maintain runbooks, playbooks, architecture diagrams, and troubleshooting guides.
• Produce regular reports on platform status, performance, alert trends, and risk areas.
• Train SOC, IR, and engineering teams on dashboards, alerting workflows, and monitoring best practices.

Stakeholder Coordination:

• Work closely with Security Operations to improve triage precision and reduce alert noise.

• Partner with the Incident Response team to enhance detection and correlation capabilities.
• Coordinate with Network Engineering to resolve sensor visibility or traffic path issues.
• Collaborate with platform owners to support upgrades, tuning cycles, and architectural enhancements.
  Required Qualifications:
• 5+ years in security operations, NDR, network engineering, or observability engineering.
• Hands-on experience with Corelight, Endace, cpacket, Zeek, Suricata, or related NDR technologies.
• Strong Python development skills, especially for API integrations and automation.
• Experience with monitoring and visualization platforms (Grafana, Prometheus, InfluxDB, Telegraf).
• Solid understanding of network traffic, packet capture, and troubleshooting.
• Ability to create dashboards, alerts, and metrics pipelines for large-scale environments.
• Experience supporting security operations teams or incident response workflows.
  
  
  
  Preferred Qualifications
• Experience developing custom Prometheus exporters (Python/Go).
• Prior exposure to Corelight APIs and Zeek script customization.
• Familiarity with Docker, Kubernetes, or containerized exporters.
• Experience with SIEM platforms and log ingestion pipelines.
• Exposure to data engineering platforms (Kafka, Elasticsearch, Loki).
• Knowledge of MITRE ATT&CK and NDR detection engineering.