Data Engineer

Technology Partners is currently seeking two engineers for an innovative AI-powered document ingestion pipeline development on Azure. Do you have experience building end-to-end data pipelines using Azure Data Factory, Azure Function Apps, and Azure OpenAI API with a focus on unstructured data and OCR processing? Let us help you make your next big career move a reality! What You Will Be Doing: You will be an integral part of a two-engineer team working within a collaborative environment to develop and maintain a sophisticated pipeline for processing unstructured documents submitted by brokers. You will focus on building, optimizing, and extending a reusable architecture that handles email ingestion, document extraction, and data landing in Snowflake and Cosmos DB. Your work will ensure the solution is auditable, scalable, and adaptable to support additional use cases and future enhancements like conversational/RAG querying. This role involves close partnership with other tech teams and requires continuous iteration to improve extraction confidence and process accuracy. This is a contract role with potential for extension based on project needs. Key Responsibilities: Develop and maintain email ingestion workflows using Azure Logic Apps, Azure Function Apps, and Azure Data Lake Storage. Implement document extraction pipelines utilizing Azure Document Intelligence and Form Recognizer for varying broker formats. Write and optimize prompts for Azure OpenAI to extract, validate, and normalize data fields. Build and support the Rules and Confidence Engine to evaluate data completeness, duplicates, and routing logic. Design and integrate Cosmos DB as the operational state store to track processing status, confidence levels, and audit logs. Create an exception handling system and automate broker clarification workflows via email alerts. Load validated structured data into Snowflake’s Silver layer through a medallion pipeline architecture. Collaborate on defining the Bronze-layer metadata tagging strategy to support future vector embedding and retrieval capabilities. Coordinate architecture efforts with other teams to prevent redundant solutions and ensure best practices. Required Skills & Experience: 3-5+ years of experience developing data pipelines and working with unstructured data. Hands‑on experience with Azure Data Factory or equivalent orchestration tools. Proficiency in Azure Function Apps using Python or C#. Strong knowledge of Azure Document Intelligence (Form Recognizer) and Azure OpenAI API, especially prompt engineering. Experience with Azure Service Bus messaging, event‑driven architecture, and Logic Apps. Familiarity with Azure Data Lake Storage Gen2 and Cosmos DB, including data modeling and partitioning. Skilled in data transformation workflows using Python and REST API integrations. Experience with Git‑based version control systems. Knowledge of Azure Key Vault for secrets management and medallion architecture principles. Prior OCR / document parsing experience, especially with unstructured formats. Bachelor’s degree in Computer Science, Information Technology, or a related field. Desired Skills & Experience: Experience with vector databases, embedding pipelines, or Azure AI Search. Familiarity with orchestration frameworks like LangChain. Knowledge of Apache Iceberg or Delta Lake table formats. Past work with insurance, financial services, or regulated industry document types. Pay: $63.00 – $90.00 /hr Keywords: Azure Data Factory, Azure Function, Azure OpenAI, OCR, unstructured data pipeline, Snowflake, Cosmos DB, Azure Logic Apps, document ingestion, AI pipeline Equal Opportunity & Background Checks: We are interested in every qualified candidate who is eligible to work in the United States. However, we are not able to provide sponsorship at this time or accept candidates who would require a corp‑to‑corp agreement. All offers of employment at Technology Partners are contingent upon clear results of a thorough background check and drug screening that meet corresponding laws and regulations at the city, state and federal level. Pay ranges are influenced by candidate qualifications, experience, and role specifics, with the actual rate determined considering skills, market conditions, and are subject to change by the employer; pay negotiations follow all state and federal legal guidelines. #J-18808-Ljbffr