Staff Security Data Engineer

About the roleAdobe’s Security Data Platform team builds and operates a petabyte-scale security data lakehouse that turns enterprise telemetry into trusted data for threat detection, investigations, compliance, and security analytics.As a Staff Security Data Engineer, you will own full-system designs and lead critical engineering initiatives across ingestion, storage, governance, orchestration, and data serving. This is a hands-on individual contributor role: you will write and review production code, build infrastructure, define test strategy, fix complex failures, and work with other engineers to deliver the architecture you help shape.You will partner with Security Operations, Threat Intelligence, Compliance, Security Engineering, and infrastructure teams to resolve technical tradeoffs and deliver reliable, cost-effective platform capabilities. Your impact will come through engineering delivery, sound architectural decisions, and helping others become stronger technical leaders.Help us evolve our team from petabyte scale toward exabyte scale!Core responsibilitiesPlatform architecture and engineeringOwn end-to-end designs that shape the Security Data Platform’s fundamental architecture on Databricks and AWS, using Spark, Delta Lake, and Unity Catalog. Translate ambiguous needs into design proposals, implementation plans, and production capabilities.Build and evolve distributed batch and streaming pipelines for EDR telemetry, network logs, identity events, cloud audit logs, and new security data sources. Design for throughput, fault tolerance, replay, schema evolution, and sub-minute freshness where use cases require it.Write maintainable Python and SQL, develop complex platform features, and refactor unreliable code. Establish reusable engineering patterns, lead code and design reviews, and own test strategy and automation choices across pipelines and infrastructure.Build and maintain tested, reusable Terraform modules for AWS and Databricks infrastructure, including S3, IAM, VPC networking, and Amazon Managed Workflows for Apache Airflow (MWAA). Automate delivery through version control and CI/CD.Data modeling governance and operationsDesign layered lakehouse data models and reusable normalization patterns, including Open Cybersecurity Schema Framework (OCSF) mappings where appropriate. Preserve source fidelity and make datasets useful for investigations, analytics, and detection workloads.Establish data contracts, quality checks, cataloging, classification, lineage, access controls, and retention policies with data owners. Use Unity Catalog to make data ownership and governance traceable from raw events to analyst-facing datasets.Engineer and operate Airflow and MWAA orchestration, including dependencies, retries, backfills, alerting, and automated recovery. Define service level objectives with consumers and instrument data freshness, completeness, pipeline health, and query performance.Lead diagnosis and resolution of complex production issues across Spark, storage, orchestration, and cloud infrastructure. Improve incident response, runbooks, recovery testing, and operational readiness to prevent recurring failures.Performance and technical leadershipProfile workloads and lead measurable cost and performance improvements through query and Spark tuning, storage lifecycle policies, partitioning, and compute right-sizing. Establish baselines and dashboards that show savings alongside reliability, latency, and capacity.Lead cross-team technical discussions, reconcile competing requirements, and build agreement on architecture and delivery priorities without formal management authority. Communicate decisions, landmarks, risks, and business impact proactively to your manager and partners.Partner with detection and security analytics engineers to deliver the data interfaces, governance, and pipeline capabilities needed for SIEM modernization and automated detection deployment.Mentor engineers through pairing, technical reviews, documentation, and knowledge sharing. Help others lead technical work, promote inclusive debate, and contribute to hiring decisions through technical assessment.Evaluate and introduce useful platform innovations, including AI-assisted engineering and operational automation. Validate outputs, protect sensitive security data, measure improvements, and help partner teams adopt proven approaches.Required qualificationsSubstantial experience building and operating production data platforms, typically 10 or more years in data or software engineering, with demonstrated Staff-level scope and impact. Equivalent depth and outcomes are welcome; a prior Staff or Principal title is not required.A record of personally delivering complex distributed systems and leading full-system design and implementation across team boundaries. Ability to turn general, unstructured assignments into clear technical approaches and working software.Deep production expertise in Spark's traditional and real-time data workloads, Databricks, Delta Lake, and lakehouse architecture; strong Python and advanced SQL skills for data modeling, profiling, and query optimization. Strong AWS experience across storage, compute, networking, and IAM, plus production Terraform experience designing, testing, and maintaining reusable infrastructure modules.Practical understanding of distributed systems, including partitioning, replication, consistency, back pressure, scalability, and fault tolerance, with experience diagnosing failures across system boundaries.Experience with Airflow, MWAA, or equivalent orchestration, and with production observability, data quality, automated testing, CI/CD, and reliable deployment and recovery practices.Demonstrated ability to improve platform cost, performance, and reliability with measurable results; sound judgment in balancing these goals against security and consumer requirements.Effective technical communication and multi-functional influence, including leading design reviews, resolving disagreements, mentoring engineers, and building adoption of shared engineering practices.Preferred qualificationsExperience with security telemetry, OCSF or comparable security schemas, and data requirements for SIEM, SOAR, threat detection, or investigations.Experience with Unity Catalog governance, data classification and retention, regional data controls, or multi-region platform design.Experience with Cribl Stream, Vector, or similar telemetry routing and enrichment tools, and platform integrations supporting automated threat detection rollout. About AdobeAdobe empowers everyone to create through innovative platforms and tools that unleash creativity, productivity and personalized customer experiences. Adobe’s industry-leading offerings including Adobe Acrobat Studio, Adobe Express, Adobe Firefly, Creative Cloud, Adobe Experience Platform, Adobe Experience Manager, and GenStudio enable people and businesses to turn ideas into impact, powered by AI and driven by human ingenuity.Our 30,000+ employees worldwide are creating the future and raising the bar as we drive the next decade of growth. We’re on a mission to hire the very best and believe in creating a company culture where all employees are empowered to make an impact. At Adobe, we believe that great ideas can come from anywhere in the organization. The next big idea could be yours. Let’s Adobe togetherAt Adobe, we believe in creating a company culture where all employees are empowered to make an impact. Learn more about Adobe life, including our values and culture, focus on people, purpose and community, Adobe for All, comprehensive benefits programs, the stories we tell, the customers we serve, and how you can help us advance our mission of empowering everyone to create.Adobe is proud to be an Equal Employment Opportunity employer. We do not discriminate based on gender, race or color, ethnicity or national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, or any other protected characteristic. Learn more.Adobe aims to make our Careers website and recruiting process accessible to any and all users. If you have a disability or special need that requires accommodation to navigate our website or complete the application process, email View email address on click.appcast.io Use Guidelines for Interviews:Our interviews are designed to reflect your own skills and thinking. The use of AI or recording tools during live interviews is not permitted unless explicitly invited by the interviewer or approved in advance as part of a reasonable accommodation. If these tools are used inappropriately or in a way that misrepresents your work, your application may not move forward in the process.At Adobe, we empower employees to innovate with AI — and we look for candidates eager to do the same. As part of the hiring experience, we provide clear guidance on where AI is encouraged during the process and where it’s restricted during live interviews. See how we think about AI in the hiring experience.Expected Pay Range:Our compensation reflects the cost of labor across several  U.S. geographic markets, and we pay differently based on those defined markets. The U.S. pay range for this position is $159,200 - $301,600 annually. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience. Your recruiter can share more about the specific salary range for the job location during the hiring process. In California, the pay range for this position is $208,300 - $301,600 At Adobe, for sales roles starting salaries are expressed as total target compensation (TTC = base + commission), and short-term incentives are in the form of sales commission plans. Non-sales roles starting salaries are expressed as base salary and short-term incentives are in the form of the Annual Incentive Plan (AIP).In addition, certain roles may be eligible for long-term incentives in the form of a new hire equity award.State-Specific Notices:California:Fair Chance OrdinancesAdobe will consider qualified applicants with arrest or conviction records for employment in accordance with state and local laws and “fair chance” ordinances.Colorado:Application Window NoticeIf this role is open to hiring in Colorado (as listed on the job posting), the application window will remain open until at least the date and time stated above in Pacific Time, in compliance with Colorado pay transparency regulations. If this role does not have Colorado listed as a hiring location, no specific application window applies, and the posting may close at any time based on hiring needs.Massachusetts:Massachusetts Legal NoticeIt is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.SummaryLocation: San Jose; San FranciscoType: Full time