DataOps Engineer

Job Summary

The DataOps Engineer is responsible for designing, building, and operationalizing data infrastructure that powers the organization's analytics and business intelligence capabilities. This role sits at the intersection of data engineering, cloud architecture, and DevOps - owning end-to-end data pipelines, platform reliability, data quality, and cloud infrastructure in a regulated environment. Responsibilities span data pipeline development, cloud migrations, platform operations, and infrastructure management across multi-cloud environments.

The ideal candidate is fluent in hands on coding of Python or PowerShell , experienced with infrastructure-as-code tools ( Terraform , CloudFormation , Ansible ), and comfortable owning both data engineering and cloud infrastructure responsibilities in a cleared, compliance-driven environment. This is a remote position based in the US requiring the ability to obtain a US Government Clearance.

Key Responsibilities

Cloud Migration & Infrastructure

• Lead the end-to-end migration of data workloads from AWS Commercial to AWS GovCloud , including S3 , Glue , Redshift , IAM , VPC , Lambda , and CloudWatch .
• Identify and resolve service parity gaps between commercial and GovCloud environments; validate data integrity, encryption posture, and access controls post-migration.
• Set up and administer enterprise Git repositories ( GitHub , Azure DevOps , or GitLab ) for the data and analytics team, including branching strategy, access controls, and code review workflows.
• Migrate existing scripts, pipelines, and artifacts into version-controlled repositories and enforce DataOps source control best practices across the team.
• Ensure all migrated and provisioned resources comply with NIST 800-171 , DFARS , CUI , and ITAR/EAR handling requirements.
• Administer and optimize cloud data services across AWS GovCloud ( S3 , Glue , Redshift , Lambda ) and Azure Government ( Synapse , ADF , ADLS Gen2 , Databricks ).
• Manage infrastructure-as-code for all data platform resources using Terraform and CloudFormation (required); Ansible preferred for configuration management.
• Monitor cloud resource utilization, enforce cost governance, and implement right-sizing recommendations for data workloads.
• Maintain backup, disaster recovery, and business continuity procedures for all data platform services.
• Serve as Tier 3 support for complex cloud infrastructure and data incidents; perform root cause analysis and implement preventive measures.

Data Pipeline Development & Orchestration

• Design, build, and maintain scalable ELT/ETL pipelines using Microsoft Fabric Data Factory , Azure Data Factory , Apache Airflow , and DBT .
• Architect end-to-end data flows from ingestion through transformation to serving layers, ensuring reliability, lineage, and observability.
• Implement real-time and batch data streaming using Apache Kafka , Azure Event Hubs , and Fabric Eventstream .
• Automate pipeline monitoring, alerting, and self-healing mechanisms to minimize manual intervention.

Microsoft Fabric Platform Ownership

• Serve as a platform engineer for Microsoft Fabric , including OneLake , Lakehouse , Warehouse , Data Factory , and Fabric Notebooks .
• Design and govern OneLake architecture including workspace topology, shortcut strategy, and medallion ( Bronze/Silver/Gold ) layer design.
• Integrate Fabric with enterprise systems including Microsoft 365 GCC High , Azure Government , Dynamics 365 , and third-party data sources.
• Configure Fabric capacity management, workspace permissions, and security in alignment with NIST 800-171 and DFARS requirements.

Data Architecture & Modeling

• Lead the design of enterprise data models including star schema , snowflake schema , and Data Vault 2.0 patterns.
• Architect Lakehouse , data warehouse, and data mesh topologies to serve analytical, operational, and reporting workloads.
• Define and enforce naming conventions, semantic layer standards, and reusable data assets using Power BI semantic models and Fabric Direct Lake .
• Collaborate with stakeholders to translate business requirements into scalable data architecture solutions.

DataOps Practices & CI/CD

• Implement DataOps principles: automated testing, version-controlled pipelines, and continuous delivery for data assets.
• Build and maintain CI/CD pipelines ( Azure DevOps , GitHub Actions ) for deploying data pipelines, models, and Fabric artifacts.
• Write and maintain automation scripts in Python and/or PowerShell to support migration, provisioning, and operational workflows.
• Containerize and orchestrate workloads using Docker and Kubernetes (AKS) for portable, scalable processing.

Data Quality & Governance

• Implement data quality frameworks including automated profiling, validation rules, anomaly detection, and SLA monitoring.
• Manage data lineage, metadata cataloging, and access governance using Microsoft Purview .
• Enforce data classification policies for CUI , ITAR-controlled , and export-sensitive data across all pipeline stages.
• Partner with compliance and cybersecurity teams to ensure data governance controls satisfy audit and regulatory requirements.

Monitoring, Troubleshooting & Support

• Proactively monitor pipeline health, data freshness, and system performance using Fabric monitoring hub , Azure Monitor , and Log Analytics .
• Document incident response procedures, runbooks, and architecture decisions (ADRs).

Collaboration & Documentation

• Partner with data analysts, data scientists, and business stakeholders to deliver reliable data products that drive decision-making.
• Create and maintain thorough documentation for pipeline design, data dictionaries, security controls, and compliance evidence.
• Participate in architecture reviews, sprint planning, and cross-functional data governance committees.

Technical Skills

Cloud Platforms

• AWS GovCloud: S3 , Glue , Redshift , IAM , VPC , Lambda , CloudWatch - including commercial-to-GovCloud migration experience
• Azure Government: ADLS Gen2 , Azure Synapse Analytics , Azure Data Factory , Azure Databricks , Azure Monitor , Microsoft Purview
• Compliance Configurations: NIST 800-171 , DFARS , FedRAMP , CUI/ITAR/EAR handling across both cloud environments

Infrastructure as Code & Automation

• Required: Terraform , AWS CloudFormation
• Preferred: Ansible - configuration management and environment automation
• Supporting: Bicep / ARM Templates for Azure Government resources

Scripting & Programming (Required - Python or PowerShell)

• Python - pandas, PySpark, SQLAlchemy, pipeline scripting, automation
• PowerShell - infrastructure automation, Azure/AWS resource management, scripting
• T-SQL / Spark SQL - complex query optimization, window functions, CTEs
• PySpark / Scala - distributed data processing on Databricks and Fabric Spark
• KQL (Kusto Query Language) - log analytics and Fabric KQL databases

Source Control & DevOps

• Git - repository setup, branching models, access management
• CI/CD: Azure DevOps Pipelines , GitHub Actions , GitLab CI
• Containers: Docker , Kubernetes (AKS)

Microsoft Fabric & Azure Ecosystem

• Core Platform: Microsoft Fabric - OneLake , Lakehouse , Warehouse , Data Factory , Eventstream , Fabric Notebooks , KQL Database ; Direct Lake mode, Fabric capacities , workspace governance
• Supporting: Azure Data Lake Storage Gen2 , Azure Synapse Analytics , Azure Databricks , Azure Event Hubs , Azure Monitor , Microsoft Purview ; Azure Government & Microsoft 365 GCC High compliance configurations

Data Pipeline & Orchestration Tools

• Apache Airflow - DAG design, dependency management, sensor patterns
• dbt (data build tool) - transformation logic, testing, documentation, incremental models
• Apache Kafka / Azure Event Hubs - real-time streaming architecture
• Talend , Informatica - legacy ETL/ELT integration patterns

Data Architecture

• Medallion architecture ( Bronze / Silver / Gold Lakehouse layers)
• Star schema , snowflake schema , Data Vault 2.0
• Data mesh and domain-oriented data ownership patterns
• Semantic layer design ( Power BI semantic models , calculation groups)

Database & Storage Systems

• Relational: PostgreSQL , Microsoft SQL Server , Azure SQL
• NoSQL: MongoDB , Cassandra , Redis , Cosmos DB
• Cloud Storage: ADLS Gen2 , AWS S3 , OneLake

Data Quality & Governance

• Microsoft Purview - data catalog, lineage, classification, policy
• Great Expectations , DBT tests - automated data quality validation
• Data classification and handling for CUI , ITAR , and EAR-controlled information

Required Qualifications

• Must have extensive experience leading end-to-end migration of data workloads from AWS Commercial to AWS GovCloud , including S3 , Glue , Redshift , IAM , VPC , Lambda , and CloudWatch .
• Must be able to demonstrate your ability to script & Program using Python or PowerShell .
• Travel, Physical and/or Government Mandated Requirements:
  • Travel up to 10-15% , including the possibility of overnights to support customer reviews and test events
  • U.S. Citizenship required due to federal contract eligibility
  • Must have or be able to obtain and maintain an active DoD Secret Clearance
  • Must comply with ITAR regulations

Education & Experience

• Bachelor's degree in computer science, Information Technology, Data Engineering, or a related field; or equivalent hands-on experience.
• 8+ years of data engineering, cloud infrastructure, or DataOps experience in a cloud-first environment.
• Demonstrated hands-on experience migrating workloads to or within AWS GovCloud or Azure Government strongly preferred.
• 2+ years of hands-on experience with Microsoft Fabric or Azure Synapse Analytics .
• Demonstrated experience in regulated environments (aerospace, defense, government contracting) with NIST 800-171 and DFARS compliance.
• Proven ability to architect and deliver enterprise-grade data platforms from greenfield to production.

Soft Skills

• Comfortable owning both data and infrastructure responsibilities - a generalist who can go deep when needed.
• Strong ability to translate complex technical architecture into clear communication for business stakeholders.
• Collaborative mindset - experienced working across data, analytics, cybersecurity, and compliance teams.
• Detail-oriented with strong documentation habits in compliance-sensitive environments.
• Proactive problem-solver with a continuous improvement orientation toward data reliability and platform scalability.

Preferred Qualifications-Highly desired

• AWS Certified Solutions Architect - Associate or Professional
• AWS Certified Data Analytics - Specialty
• HashiCorp Certified: Terraform Associate
• Microsoft Certified: Fabric Analytics Engineer Associate (DP-600)
• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Microsoft Certified: Azure Solutions Architect Expert
• Databricks Certified Data Engineer Associate / Professional
• DBT Certified Developer
• CompTIA Security+ or CySA+