Senior Data Engineer

Senior Data EngineerLocation: Herndon, VA (Remote Work)Must have an Public Trust ClearanceKey Responsibilities• Provide authoritative expertise on data engineering methods and best practices, including code first development approaches and modern pipeline design patterns.• Design, implement, and maintain the data architecture that supports products and end users, with all assets managed under source control.• Design, implement, and maintain ELT and ETL pipelines for efficient processing of source data in Azure Synapse and Azure Machine Learning, using both SDK V1 and SDK V2.• Migrate source data identified by SBA OIG into Azure Data Lake Storage.• Normalize entity attributes such as addresses, phone numbers, and other common fields.• Review, maintain, and improve existing architecture and pipelines, including periodic audits addressing bottlenecks, deprecated dependencies, and architecture drift.• Establish quality controls across all pipelines and introduce error handling, logging mechanisms, and validation checks.• Incorporate source control across all pipelines and analytics codebases so code can evolve iteratively without destabilizing the architecture.• Optimize ingestion, processing, and storage across a wide variety of datasets and data types, including modern columnar formats such as Parquet.• Develop self service capabilities that let SBA OIG analysts query and export data for investigations and audits.• Author robust standard operating procedures governing the authoring, development, validation, publishing, execution, and monitoring of all data pipelines and assets in the Azure environment.• Produce detailed documentation of the data architecture, including data dictionaries, entity relationship diagrams, and pipeline process maps.• Maintain and expand the environment with additional datasets and services on request, following a defined intake and testing process before production deployment.• Stay current with emerging AI tooling relevant to data engineering and contribute to exploratory work evaluating automation and language model assisted capabilities.