Senior Data Engineer

Engineering – Senior Data Engineer – Atlanta or DC Area Preferred – Full-Time DNS and netflow telemetry lands in a data lake in our own AWS account, detection pipelines score it, and the findings are written into an intelligence graph. Security researchers design the detections and often build them. This team builds everything around them: the pipelines that reduce lake data to what a detector needs, the packaging that lets it run on our platform, the orchestration that runs it on schedule at full volume, and the path that writes what it found back to the lake as evidence. Our stack is Amazon EKS, Spark, Apache Iceberg and Delta Lake, Argo Workflows, Argo CD, Crossplane, and Python and Scala. We do not expect you to have used all of it. We do expect that you have worked in each of these areas, in whatever tools you used at the time: continuous integration and delivery, infrastructure as code, distributed data processing, container orchestration, a data lake on an open table format, and Python. This role gets detections into production and keeps them there. A researcher hands you a detection, sometimes as working code, sometimes as a written technique. Everything between the data lake and a finding recorded as evidence is yours. Most of the difficulty is correctness under conditions a researcher's environment never faces: the volume is far larger, the data has gaps and skew, the job will be re-run, and someone will eventually need to explain why a finding says what it says. Location Atlanta or DC Area Preferred Committment Full-Time Work style Remote Responsibilities Reduction. Detectors do not run against the whole lake. Write the Spark jobs that cut telemetry at our volume down to the slice a detector needs, at the grain it expects. Packaging. Containerize the detection so it runs the same way every time, with its dependencies pinned and its interface fixed. Orchestration. Build the workflow that runs it on schedule at full volume, survives partial failure, and can be re-run over a past window without producing duplicate findings. Write-back. Write what it found into the canonical tables as evidence, with the provenance that makes a finding defensible: which detector, which version, which window, and what data it was based on. Data quality in both directions. Verify that incoming data conforms to the schema we specified, and catch output that does not look like what the detection predicted. Required Qualifications Spark in production at volume, with real tuning experience. Skew, partitioning, shuffle behavior, join strategy. An open table format in production: merges, incremental processing, compaction and table maintenance. We run both Apache Iceberg and Delta Lake. Either one is fine. You have containerized someone else's code and run it on a schedule in production. Dependency pinning, resource limits, and what happens when it fails halfway through. Workflow orchestration in production, in any tool. Argo Workflows, Airflow, Dagster, Step Functions. We use Argo Workflows and will not test you on it. Event-time correctness. You know why processing time and event time differ and what breaks when they are conflated. You have taken something that worked in one person's environment and made it run without supervision. This is most of the job and we will ask about a specific case. Comfort working where correctness is a matter of degree rather than a passing test. Must be a US citizen Preferred Qualifications Our specific stack: Amazon EKS, Argo Workflows, Argo CD, Crossplane. Scala. Not required, though some of what you maintain is written in it. High-volume network telemetry. Why Join Us Fully remote within the United States, with flexible working hours End-to-end ownership: everything between the data lake and a recorded finding A modern stack and an engineering culture that values automation and best practices #J-18808-Ljbffr Voreas Laboratories Inc.