Data Engineer

Director, Data LeadAs the Director, Data Lead for the PGIM Product Technology, you will engage in requirements gathering, lead design, architect and partner with business owners, technology leads and engineering teams to support shift to Microsoft cloud platforms for data processing and management. This candidate will work to define and enforce standards while also overseeing end-to-end data engineering including sourcing, modeling, security, governance, and business intelligence functions. You will design and lead engineers to implement solutions in accordance with industry best practices. Your expertise in data processing and data analytics using the Microsoft Fabric platform will be crucial in delivering successful data solutions. The candidate will play a key role to bridge solutions, communication, timelines, and challenges/solutions to project stakeholders for multiple, concurrent overlapping implementation initiatives.This role is based in our office in Newark, NJ. Our organization follows a hybrid work structure where employees can work remotely and from the office, as needed, based on demands of specific tasks or personal work preferences. This position is hybrid and requires your on-site presence on a reoccurring weekly basis 3 days per week.What you can expect:Design and Implement Configurable Data Ingestion ETL framework including Data Quality, transformation and modelling. Implement Lakehouse solutions using Microsoft Fabric and paving the road to PGIM future state using Azure services and Fabric Infrastructure.Develop and deploy data integration solutions, including data ingestion, transformation, and delivery to consumers within Microsoft Fabric, while utilizing optimal storage formats and layers.Work with Infrastructure/Devops teams to come up with CI/CD pipelines as a framework that can be extended across organization.Architect solutions that incorporate BI tools like Power BI ensuring seamless data integration for analytics and reporting.Ensure data accuracy, consistency, and integrity across the data warehouse ecosystem by crafting robust solution designs.Design and Build AI solutions leveraging search, embeddings, retrieval, ranking and Retrieval-Augmented Generation (RAG).Set up roadmap for AI as a service to consuming systems and business stakeholders.Establish a workspace architecture strategy through design and implement of data warehouse and lakehouse solutions allowing data sharing between different teams avoiding duplication, yet adhering to security and governance rules.Lead a small team to design/build and monitor data and AI solutions.Ensure teams follow best practices and standards for security, compliance, scalability, reliability, and performance.Serve as SME in the Fabric space to technology partners, provide hands on training to development teams, and mentor business areas to maximize Fabric AI Capabilities.Create and maintain comprehensive documentation (Playbooks) of solution architectures to internal and external stakeholders covering Data Fabric architecture, processes, and procedures.Collaborate closely with our business and technology partners including data analysts, architects, engineers, and client stakeholders to gather requirements and translate them into scalable Microsoft Fabric solutions.Proactively seek out new knowledge and adapt to new trends, technical solutions, and patterns that improve the availability, reliability, efficiency, observability, and performance of products while also driving consistency in monitoring and operations at scale, and share knowledge with other engineers.What you will bring:Bachelor's or Master's degree in Computer Science, Data Engineering or related discipline.10+ years of experience in cloud computing with Microsoft Azure leading migrations, integrations, and deployments. 10+ years of experience as a Data Architect, with demonstrated expertise in designing, managing, and deploying client solutions.Strong knowledge of Azure services and features including Azure Synapse Analytics, Azure Data Factory, Azure Storage Accounts, Azure Logic Apps, Azure Functions, Azure SQL Database, Azure Active Directory, Azure DevOps, Azure Monitor, etc.Strong knowledge of Fabric workloads and features including Notebooks, Deployment Pipelines, Workspace architecture, Workspace Security, Data Warehouse/Lakehouse Architecture, Semantic Model, Data Factor, Data Flow, Data Activator, AI Skills, and Copilot, ultimately leading to AI capabilities in Power BI.Build and deploy intelligent applications using Azure AI Foundry, leveraging prompt orchestration, grounding with Azure AI Search, and agent orchestration features for enterprise-scale GenAI use cases. Independently implement GenAI workflows using Foundry.Proficient in CI/CD pipelines to deploy in azure environment using GitHub Actions.Proven ability to implement data governance tools (Purview preferred), data security (OneLake Security preferred), and compliance requirements like data retention schedules and audit.Specific experience in .Net architecture, design, and development using Azure platform services. Must have the proven ability of delivering multiple concurrent, overlapping projects in a fast paced technology team on-time, on budget and within scope.Experience in leading technical teams to drive business outcomes.Financial Services experience working with Asset Management esp on Institutional funds and Mutual funds preferred.Must come to this position with superior communication, collaboration, and problem-solving skills capable of conveying technical concepts to both technical and non-technical stakeholders.Microsoft certifications in data-related fields are preferred, such as Azure Solutions Architect Expert, Azure Developer Associate, Azure Administrator Associate, Azure Data Engineer, Fabric Engineer, Power BI Data Ana*PGIM welcomes all applicants, even if you don't meet every requirement. If your skills align with the role, we encourage you to apply.Note: Prudential is required by state specific laws to include the salary range for this role when hiring a resident in applicable locations. The annual base salary range for this role is from $170,000 to $225,000. Specific pricing for the role may vary within the above range based on many factors including geographic location, candidate experience, and skills.