Data Engineer

Overview Technology Partners is currently seeking two engineers to develop an AI-powered document ingestion pipeline on Azure. The role focuses on end-to-end data pipelines using Azure Data Factory, Azure Function Apps, and the Azure OpenAI API, with an emphasis on unstructured data and OCR processing. What You Will Be Doing You will be an integral part of a two-engineer team in a collaborative environment to develop and maintain a pipeline for processing unstructured documents submitted by brokers. The work includes building, optimizing, and extending a reusable architecture that handles email ingestion, document extraction, and data landing in Snowflake and Cosmos DB. The solution should be auditable, scalable, and adaptable to support additional use cases and future enhancements such as conversational/RAG querying. This role involves close partnership with other tech teams and requires continuous iteration to improve extraction confidence and process accuracy. This is a contract role with potential for extension based on project needs. Key Responsibilities Develop and maintain email ingestion workflows using Azure Logic Apps, Azure Function Apps, and Azure Data Lake Storage. Implement document extraction pipelines utilizing Azure Document Intelligence and Form Recognizer for varying broker formats. Write and optimize prompts for Azure OpenAI to extract, validate, and normalize data fields. Build and support the Rules and Confidence Engine to evaluate data completeness, duplicates, and routing logic. Design and integrate Cosmos DB as the operational state store to track processing status, confidence levels, and audit logs. Create an exception handling system and automate broker clarification workflows via email alerts. Load validated structured data into Snowflake’s Silver layer through a medallion pipeline architecture. Collaborate on defining the Bronze-layer metadata tagging strategy to support future vector embedding and retrieval capabilities. Coordinate architecture efforts with other teams to prevent redundant solutions and ensure best practices. Required Skills & Experience 3-5+ years of experience developing data pipelines and working with unstructured data. Hands-on experience with Azure Data Factory or equivalent orchestration tools. Proficiency in Azure Function Apps using Python or C#. Strong knowledge of Azure Document Intelligence (Form Recognizer) and Azure OpenAI API, especially prompt engineering. Experience with Azure Service Bus messaging, event-driven architecture, and Logic Apps. Familiarity with Azure Data Lake Storage Gen2 and Cosmos DB, including data modeling and partitioning. Skilled in data transformation workflows using Python and REST API integrations. Experience with Git-based version control systems. Knowledge of Azure Key Vault for secrets management and medallion architecture principles. Prior OCR / document parsing experience, especially with unstructured formats. Bachelor’s degree in Computer Science, Information Technology, or a related field. Desired Skills & Experience Experience with vector databases, embedding pipelines, or Azure AI Search. Familiarity with orchestration frameworks like LangChain. Knowledge of Apache Iceberg or Delta Lake table formats. Past work with insurance, financial services, or regulated industry document types. Pay $63.00 – $90.00 /hr Additional Information We are restricting sponsorship to eligible United States workers and cannot offer corp-to-corp arrangements at this time. All offers of employment are contingent upon the results of a background check and drug screening conducted in accordance with applicable laws. Pay ranges are influenced by candidate qualifications, experience, and role specifics, and are subject to change by the employer; pay negotiations follow all state and federal legal guidelines. #J-18808-Ljbffr