Senior Azure Data Migration & Dataverse Integration Engineer

Role: Senior Azure Data Migration & Dataverse Integration Engineer Company: Arch Systems Location: Remote Type: Full-time Position Summary Seeking a Senior Azure Data Migration & Dataverse Integration Engineer to design, develop, implement, and support enterprise data migration and synchronization capabilities between Microsoft SQL Server and Microsoft Dataverse . The position requires a senior hands-on engineer capable of owning the complete data pipeline lifecycle, including source analysis, data mapping, extraction, transformation, migration, synchronization, validation, exception handling, monitoring, performance optimization, and automated deployment. The engineer will develop and maintain repeatable pipelines supporting initial bulk migration, incremental migration, Change Data Capture (CDC), ongoing synchronization, and transition/cutover activities between legacy SQL Server applications and modern Microsoft Power Platform/Dataverse solutions. Primary Responsibilities The Data Migration Engineer will: Design and implement end-to-end SQL Server-to-Dataverse data migration pipelines . Develop automated pipelines using Azure Data Factory (ADF) and supporting Azure services. Analyze legacy SQL Server databases, tables, views, relationships, stored procedures, and data dependencies. Map legacy SQL Server data structures to Dataverse tables, columns, relationships, choices, lookups, and alternate keys. Develop repeatable processes for: Initial/bulk data migration Incremental data migration Change Data Capture Delta processing Scheduled synchronization Near-real-time synchronization Legacy-to-Dataverse synchronization Dataverse-to-legacy synchronization where required Cutover and final synchronization Develop and maintain ADF pipelines, datasets, linked services, integration runtimes, triggers, parameters, variables, and reusable pipeline components. Develop transformation and data-cleansing logic required to convert legacy data into Dataverse-compatible structures. Implement controlled sequencing to maintain parent/child relationships and referential integrity. Develop lookup resolution and cross-reference processes for Dataverse GUIDs and legacy SQL identifiers. Implement restartable and recoverable migration processes. Develop pipeline logging, audit, exception handling, retry, and reconciliation capabilities. Validate migrated data for completeness, accuracy, integrity, and consistency. Perform source-to-target record-count and field-level reconciliation. Troubleshoot failed records, pipeline failures, API limitations, data-quality issues, and performance problems. Develop automated deployment processes for data pipeline components across DEV, TEST, UAT, and PROD environments. Azure Data Factory / Pipeline Responsibilities The candidate must have advanced hands-on experience designing and developing Azure Data Factory solutions. Responsibilities include: ADF pipeline architecture and development Copy Activities Lookup Activities Stored Procedure Activities ForEach and Until processing Conditional processing Pipeline dependencies Parameterized pipelines Dynamic expressions Metadata-driven pipeline development Linked Services Datasets Integration Runtime configuration Managed Identity authentication Azure Key Vault integration Pipeline triggers and scheduling Incremental load processing Watermark processing CDC patterns Retry and failure handling Pipeline logging and auditing Performance tuning and parallelization Environment-specific configuration Infrastructure and pipeline deployment automation The engineer should be capable of developing reusable, metadata-driven pipeline patterns rather than creating independent hard-coded pipelines for every table. SQL Server Responsibilities Strong SQL Server development and data engineering skills are required, including: Complex T-SQL development Stored procedures Views Common Table Expressions Temporary and staging tables Data profiling Data cleansing Data transformation Joins and relationship analysis Primary and foreign key analysis Indexing and query optimization SQL Server Change Data Capture Change Tracking Watermark-based incremental extraction Data validation queries Reconciliation queries Duplicate detection Referential-integrity validation The engineer must be capable of analyzing legacy databases with limited documentation and determining the technical relationships necessary to migrate data successfully. Microsoft Dataverse Responsibilities Advanced Dataverse knowledge is required, including: Dataverse tables and columns Standard and custom tables Table relationships One-to-many and many-to-many relationships Lookups Choice columns Alternate keys GUID management Ownership and security considerations Dataverse Web API Dataverse APIs and connectors Upsert patterns Batch processing Duplicate detection Dataverse service protection/API limits Application users and service principals Environment configuration Solutions and ALM The candidate must understand how Dataverse differs from a traditional relational SQL database and design migration processes accordingly. Data Synchronization The engineer will design synchronization processes supporting coexistence between legacy and modernized applications during phased transition. This includes: SQL Server → Dataverse synchronization Dataverse → SQL Server synchronization where required Unidirectional and bidirectional synchronization patterns Incremental synchronization CDC and watermark processing Event-driven integration where applicable Conflict detection and resolution Source-of-record determination Duplicate prevention Idempotent processing Failed-record reprocessing Synchronization audit history Data latency monitoring Reconciliation between source and target systems The engineer must understand the risks associated with bidirectional synchronization, including update conflicts, synchronization loops, transaction ordering, and determination of the authoritative system of record. Data Migration Architecture The candidate should be capable of implementing a controlled migration architecture such as: Legacy SQL Server → Extraction → Staging/Bronze → Validation/Transformation → Silver → Business Rules/Mapping → Gold/Target-Ready Data → ADF Load Pipeline → Dataverse → Reconciliation The architecture must support traceability from the original source record through transformation and final Dataverse record creation. Pipeline Monitoring and Operations Develop operational capabilities to monitor migration and synchronization processing, including: Pipeline execution status Records extracted Records transformed Records successfully loaded Records rejected Warning and error counts Source and target record counts Processing duration Synchronization latency Retry status Failed-record queues Reconciliation status Implement monitoring using applicable Microsoft technologies such as: Azure Monitor Application Insights Log Analytics ADF monitoring Azure SQL logging Power BI or operational dashboards DevOps and CI/CD The engineer will establish controlled deployment and configuration management for all data migration components. Responsibilities include: Azure DevOps Repos Branching and source-control practices Pull requests and code reviews Azure DevOps Pipelines Automated DEV/TEST/UAT/PROD deployments Environment-specific parameters Infrastructure as Code Bicep and/or ARM templates ADF deployment automation SQL database deployment Dataverse configuration coordination Key Vault and secret-management integration Release validation Rollback procedures All pipelines, scripts, SQL objects, infrastructure definitions, configuration, and supporting code must be maintained under source control. Security Requirements The engineer will implement data migration capabilities consistent with enterprise security requirements, including: Microsoft Entra ID Managed Identities Service Principals Dataverse Application Users Role-Based Access Control Least-privilege access Azure Key Vault Secure connection management Encryption in transit and at rest Audit logging Environment separation Controlled production access Experience supporting Federal Government, DoD, RMF, NIST 800-53, or comparable regulated environments is preferred. Required Technical Skills 7+ years of data engineering, database development, integration, or data migration experience. 4+ years of Microsoft Azure data engineering experience. Advanced Azure Data Factory development experience. Advanced Microsoft SQL Server and T-SQL experience. Strong Microsoft Dataverse experience. Experience migrating enterprise SQL data into Dataverse. Experience with CDC, incremental loads, delta processing, and synchronization. Experience developing ETL/ELT pipelines. Experience with REST APIs and Dataverse Web API. Experience with Azure SQL. Experience with Azure Key Vault and Managed Identities. Experience with Azure DevOps source control and pipelines. Experience implementing automated data validation and reconciliation. Preferred Technical Experience Experience with several of the following is highly desirable: Azure Data Lake Storage Gen2 Azure SQL Database Azure SQL Managed Instance Microsoft Fabric Azure Functions Azure Service Bus Azure Event Grid Logic Apps Power Automate Power Apps Power BI Synapse Analytics C#/.NET Python PowerShell Bicep Git Azure DevOps Preferred Certifications One or more of the following Microsoft certifications is preferred: DP-203 – Azure Data Engineer Associate or current Microsoft equivalent PL-400 – Power Platform Developer Associate PL-600 – Power Platform Solution Architect Expert Microsoft Azure Administrator, Developer, or Solutions Architect certification Desired Candidate Profile This position requires a hands-on senior data engineer , not solely a data architect or Power Platform administrator. The successful candidate must be able to independently take a legacy SQL data requirement through the complete technical lifecycle: Legacy SQL Analysis → Data Profiling → Source-to-Target Mapping → Pipeline Design → ADF Development → Transformation → Dataverse Load → CDC/Synchronization → Validation → Reconciliation → CI/CD → Production Operations The engineer must be capable of working directly with application developers, Power Platform developers, database administrators, solution architects, business analysts, and system owners to determine how legacy data will be migrated, synchronized, validated, and ultimately transitioned from the legacy environment. Key Deliverables The position will be responsible for producing and maintaining: Source-to-target data mappings Data migration pipelines Data synchronization pipelines ADF pipeline framework SQL extraction and transformation scripts Staging database structures CDC/incremental processing framework Dataverse loading processes Data-quality rules Exception and retry processes Migration audit records Data reconciliation reports Pipeline monitoring CI/CD deployment pipelines Infrastructure-as-Code definitions Migration runbooks Cutover procedures Production support documentation Technical architecture and data-flow diagrams