Lead Data Engineer

Job Description AWAITING FINAL AWARD* At Nyla, we are proud to deliver solutions that make a real impact! We are looking for a visionary Lead Data Engineers to guide our engineering efforts in designing, building, and optimizing high-throughput big data solutions. In this role, you will lead automated big-data extraction and ingestion pipelines using Unity Catalog and PySpark while steering team strategy. You'll drive the core Data Observability Layer for the platform, direct technical execution, and support delivery in Enterprise Platform Data Observability, and solve complex enterprise analytic challenges. You will be building "enterprise capabilities" within the Department of War's authoritative data analytics platform (War Data Platform) for the Chief Digital and AI Officer (DoW CDAO). Key Responsibilities Lead the development and operation of end-to-end, automated data ingestion pipelines using PySpark and Databricks. Guide the team in maintaining and optimizing data structures using Delta Tables to enable seamless data versioning, backups, and rollback safeguards. Partner with Platform and Solution Architects to align architecture, connecting Delta Tables with Vector Databases while strictly enforcing Role-Based Access Controls (RBAC). Mentor engineers, oversee technical delivery, and uphold high engineering standards across data reliability and observability initiatives. The annual base salary range for this role is $160,000-$230,000 (USD), which does not include discretionary bonus compensation or our comprehensive benefits package. Actual compensation offered to the successful candidate may vary from posted hiring range based upon geographic location, work experience, education, and/or skill level, among other things. Required Skills Big Data & Pipelines: Expertise in Python, PySpark, and Databricks for building and operating end-to-end automated data ingestion pipelines. Data Architecture & Management: Hands‑on experience with Delta Lake Architecture, Delta Tables, and Unity Catalog to manage versioning, backups, and rollbacks. Automation & Querying: Strong proficiency in SQL and ETL/ELT automation practices. Access & Integration: Familiarity with Role-Based Access Controls (RBAC) and connecting data structures to Vector Databases. Clearance: Active Top Secret, SCI Eligible Experience: 10+ years of progressive professional experience in data engineering, software engineering, or a related technical discipline (Levels III and IV). Education: Bachelor’s Degree in Computer Science, Data Science, Information Technology, or a related field. Work Location: Arlington County, VA (Hybrid) Desired Skills Strong track record of technical mentorship and leading collaborative engineering efforts alongside Platform and Solution Architects. Exceptional problem-solving skills focused on driving scalable, high-throughput solutions. Passion for fostering team innovation in delivering high-impact data reliability and observability tools. About Nyla Technology Solutions Nyla Technology Solutions delivers exceptional Artificial Intelligence (AI), Data Science, and Software Engineering services for the U.S. Government. Nyla embraces a forward-thinking and bold approach at every turn, earning us a solid reputation of technical trendsetters within the industry. We have a passion for developing solutions that have a quick and immediate impact on mission. Headquartered in Columbia, Maryland, our customers love how we tackle their most challenging problems and get things done. If you have the unique experience and expertise we are seeking, along with the desire and determination to invest your time and energy as a part of Nyla’s team, Taking Care of All of You Nyla provides a top-of-market compensation and benefits package. And through our unique Nyla FLEX program, we custom tailor these benefits to best fit your lifestyle. Benefits PAY - Nyla starts with 4 weeks of Annual Leave plus 11 holidays and an additional day of Annual Leave for each year you’re at the company. You have the flexibility to cash out your annual leave hours, opt out of other Nyla benefits, and/or arrange for additional hours on contract* (over 40 hrs/week). There’s even an option to earn 1.3 times your hourly rate once you work over 1880 hours on contract! LEAVE - Want to spend more time with the family? Want more time to travel the world? You can BUY additional annual leave for a total of 6 weeks of annual leave. That’s up to 240 hours of leave plus 11 holidays! That’s not even including paid anniversary leave! SCHEDULE - Does the traditional 40-hour workweek no longer fit your lifestyle? With Nyla FLEX, you have the freedom to scale down to 30-32 hours while still enjoying the top-notch Nyla benefits you know and love. It’s flexibility that works for you without compromising the perks! WHAT ABOUT OTHER BENEFITS? Nyla’s health care (medical, dental, and vision) is 100% covered by the company. We provide 10% 401k matching - with full vesting day 1! Our Professional Development offers $5,000 per year to be used towards fees, tuition, or time off for your continued growth. We even have a student loan repayment program and we provide 8 hours of volunteering annually so you can support your community, making your world a better place. Nyla is an equal opportunity employer. #J-18808-Ljbffr