Data Engineer

Data EngineerArlington, VA540 is seeking a Data Engineer to support a mission-critical technology modernization effort for the Department of War. You will design, build, and maintain Databricks-based data pipelines and lakehouse capabilities that enable secure data integration, analytics, AI/ML, and operational workloads at enterprise scale.Working with software engineers, AI/ML engineers, cybersecurity teams, and mission stakeholders, you will build scalable and reliable solutions using Databricks, Python, Apache Spark, and Delta Lake. The ideal candidate enjoys solving complex engineering challenges and developing trusted data products that support national defense missions.Location: Arlington, VA Citizenship & Clearance Requirement: Per client requirements, candidates must be U.S. Citizens with an active DoW Secret (or higher) clearance Education Requirement: Bachelor's degree in Computer Science, Engineering, or a related technical field preferred; equivalent combinations of education and relevant experience will be considered 540 Internal Thrive Level: Data Engineer II or IIIWHY 540?540 is a forward-thinking company that the government turns to in order to getshitdone. We don't just talk about innovation – we deliver it. We break down barriers, build impactful technology, and solve mission-critical problems.HOW YOU'LL DRIVE IMPACTDesign, develop, and maintain Databricks-based data pipelines, data products, and lakehouse capabilitiesBuild automated ETL/ELT pipelines that ingest, transform, and deliver mission-critical dataDevelop production-grade data-processing solutions using Python, SQL, PySpark, Apache Spark, and Delta LakeDesign and maintain data models, schemas, tables, and medallion architecture patterns supporting analytical, operational, and AI/ML workloadsBuild and operate batch and streaming data pipelines supporting mission requirementsDevelop and manage Databricks notebooks, jobs, workflows, clusters, and compute resourcesImplement data-quality checks, automated testing, monitoring, lineage, and metadata-management capabilitiesSupport data discovery, governance, and access controls using Unity Catalog or similar technologiesOptimize Spark workloads and Databricks resources for performance, scalability, reliability, and cost efficiencyCollaborate with engineers, analysts, and data scientists to deliver reusable data products and mission capabilitiesSupport Databricks deployments using CI/CD, infrastructure as code, and source controlPartner with cybersecurity teams to implement data-protection, access-control, auditing, and governance requirementsTroubleshoot issues affecting Databricks workloads, data pipelines, storage systems, and production data servicesDocument data models, pipeline designs, engineering processes, and operational proceduresREQUIRED SKILLS & EXPERIENCE4+ years of relevant data engineering or software engineering experienceHands-on experience developing and operating production data pipelines using DatabricksProficiency with Python, SQL, PySpark, Apache Spark, and Delta LakeExperience building automated ETL/ELT pipelines for large-scale datasetsExperience designing and maintaining data models, schemas, tables, and lakehouse architecturesExperience managing Databricks notebooks, jobs, workflows, and compute resourcesExperience implementing data quality, automated testing, monitoring, lineage, or metadata-management capabilitiesExperience working with Databricks and cloud-based data services in AWS, Azure, or Google CloudExperience working with structured, semi-structured, and unstructured dataUnderstanding of lakehouse architecture, data governance, security, privacy, and access-control principlesAbility to troubleshoot data pipelines, Spark workloads, infrastructure, and applicationsNICE TO HAVEDatabricks certification or equivalent demonstrated platform expertiseExperience supporting DoW, federal, Advana, or other enterprise data environmentsExperience with CI/CD, infrastructure as code, automated testing, and source controlExperience using Unity Catalog for data governance, lineage, and access controlExperience developing streaming pipelines with Spark Structured Streaming, Kafka, Kinesis, or PulsarExperience with orchestration tools such as Airflow, Dagster, or Argo WorkflowsExperience with Docker, Kubernetes, or other containerization and orchestration technologiesExperience building cloud-native data platforms in secure, regulated, classified, or mission-critical environmentsCurrently holds, or is willing to obtain within 30 days of employment, an approved certification such as Cloud+, GSEC, Security+, or SSCPBENEFITS & PERKSFlexible PTO + all Federal holidays offHealth, dental and vision insurance plansFlexible Spending Account (FSA)401k with employer matchCompany-sponsored life insurance, short- and long-term disabilityProfessional development (training, certifications, conferences)Paid cloud developer accountsReferral bonusesHQ office perks (parking / metro reimbursement, nitro coffee & lunches)Annual social events (540 Week, hackathon, charity golf tournament, etc.)Access to 540's Washington Capitals & Nationals ticketsEQUAL EMPLOYMENT OPPORTUNITY (EEO)540's policy is to provide equal employment opportunity to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.