Data Integration Engineer

Data Integration Engineer Primary Responsibilities

• Integration Development: Design, build, and maintain integration pipelines connecting source systems to downstream targets, using containerized Python services on Azure Container Apps with Service Bus and Event Hub.
• Event-Driven and Streaming Integration: Build and operate flows against Salesforce's Pub/Sub API over gRPC, including Platform Event publish and subscribe, Avro decode, durable replay checkpointing, and reconnection handling for long-lived subscriptions.
• Reliability and Failure Handling: Implement idempotent processing, deduplication, retry policies, and dead-letter consumers so at-least-once delivery does not produce duplicate or lost records in target systems.
• API Design and Authentication: Design and consume REST APIs, including OAuth 2.0 with token caching, rate limit handling, and error semantics that distinguish retryable from terminal failures.
• Transformation and Business Logic: Build and maintain Python-based transformation and validation services that enforce business rules as data moves between systems.
• Data Platform Integration: Land request and response event streams into Snowflake via Event Hub as a durable parallel stream, so integration activity is available for reporting, reconciliation, and audit.
• Observability and Handoff to Operations: Instrument integrations for distributed tracing, structured logging, and correlation across asynchronous hops, and deliver monitoring hooks, alert thresholds, runbooks, and escalation paths to Operations before go-live.
• Testing, CI/CD, and Requirements: Write unit and integration tests covering failure and replay paths, maintain CI/CD pipelines (e.g., GitHub Actions) across DEV, SIT, UAT, and PROD, and translate stakeholder requirements into technical designs, including which pattern applies.