Data Engineer

Data EngineerKDA Consulting Inc. is seeking a highly skilled Data Engineer to design, build, and maintain scalable data pipelines and architectures supporting mission-critical programs within the Intelligence Community (IC). This role focuses on enabling efficient data processing, analytics, and decision-making by delivering reliable, high-quality data solutions in cloud environments.The ideal candidate will have strong experience with AWS cloud services, data engineering best practices, and big data technologies, with the ability to operate in a fast-paced, collaborative environment supporting complex mission systems.Data Pipeline Development & EngineeringDesign, develop, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from multiple sourcesBuild and optimize data workflows supporting both batch and real-time processingEnsure high availability and reliability of data pipelines across enterprise systemsAWS Cloud & Data ServicesLeverage AWS services such as S3, Glue, Lambda, Redshift, EMR, and Kinesis to support data engineering solutionsDesign and manage cloud-based data architectures that are secure, scalable, and cost-efficientImplement best practices for data storage, partitioning, and lifecycle managementData Optimization & PerformanceOptimize data storage and retrieval for performance, scalability, and cost efficiencyImplement indexing, partitioning, and query optimization strategiesMonitor and troubleshoot data pipeline performance issuesData Quality & GovernanceEnsure data accuracy, consistency, and integrity across all data pipelines and systemsImplement validation, monitoring, and error-handling mechanismsSupport data governance practices and compliance with security and IC standardsCollaboration & IntegrationWork closely with data scientists, software engineers, and mission stakeholders to support analytics and operational needsIntegrate data solutions with enterprise applications, APIs, and downstream analytics platformsParticipate in Agile development processes and contribute to continuous improvement efforts