Databricks Architect

Databricks Architect

PEMCO is seeking a Databricks Architect to join our community. The Databricks Architect leads the end-to-end architecture, governance, and scalability of the Azure Databricks Lakehouse platform at PEMCO. The Databricks Architect is responsible for landing zone design, security and compliance, data governance with Unity Catalog, performance and cost optimization, and cross environment (Dev/Test/Prod) standards. The architect partners closely with Enterprise Architecture, Security, Networking, Data Engineering, Analytics, and Product teams to deliver a secure, reliable, and cost efficient platform that accelerates use cases across underwriting, claims, pricing, customer engagement, fraud, and enterprise analytics.

What You'll Be Doing:

• Own the Azure Databricks landing zone architecture: subscription and resource group strategy, workspace hierarchy, VNet injection, Private Link/endpoints, firewall rules, and cross region DR/BCP approaches.
• Define and implement Lakehouse patterns (bronze/silver/gold) using Delta Lake, including CDC ingestion with Auto Loader, streaming with Structured Streaming and Delta Live Tables (DLT), and table optimization (OPTIMIZE/Z ORDER/VACUUM policies).
• Establish and govern Unity Catalog: metastore design, external locations, schema/catalog standards, lineage integration, access models (RBAC/ABAC), and data protection (row/column level security, masking, tags).
• Create platform governance standards: SCIM provisioning, SSO/SAML, secret management (Key Vault backed scopes), cluster and pool policies, IP access lists, workspace and job ACLs, repo standards, and promotion workflows across Dev/Test/Prod.
• Build CI/CD and release engineering for Databricks (Repos, Git integration, ADO/GitHub Actions, Terraform/Bicep/ARM), including environment promotion, approvals, and automated quality gates.
• Implement observability and reliability practices: platform health dashboards, UC audit log analytics, job/runtime metrics, event logs, incident response runbooks, SLO/SLI reporting, and capacity planning.
• Drive performance engineering: benchmark Spark workloads, Photon adoption, caching strategies, file sizing and partitioning, concurrency controls, and SQL warehouse tuning.
• Lead FinOps for Databricks: budgets and tagging strategy, cluster policy guardrails (autoscale, instance families, spot usage, idle timeouts), serverless/SQL cost controls, and unit economics reporting.
• Partner with Security & Compliance to meet enterprise and regulatory requirements (encryption at rest/in transit, key management, PII handling, audits, retention policies, vulnerability management).
• Integrate the platform with Azure and enterprise services: ADLS Gen2, Event Hubs/Kafka, AAD, Purview, Key Vault, Log Analytics, Tableau, Salesforce, Genesys, and Delta Sharing.
• Enable MLOps and analytics at scale: MLflow model registry standards, feature engineering conventions, real time/batch inference patterns, Workflows orchestration, and blue/green or canary deployment strategies.
• Coach, guide, and review solution designs from data engineers and analysts; conduct architecture/design reviews and code reviews; publish reference architectures, standards, and reusable templates.
• Develop and maintain platform documentation, onboarding materials, and runbooks; lead knowledge transfer and internal training.
• Lead design and rollout of Databricks Mosaic AI (Foundation Model APIs, Model Serving, Vector Search, and Agent Framework) for enterprise GenAI use cases (RAG over governed data, copilots, and agents).
• Define prompt/response patterns, evaluation frameworks, guardrails (safety filters, PII redaction), offline/online tests, latency/error SLOs, and cost controls.
• Build LLMOps on the Lakehouse: instrument MLflow/Unity Catalog–backed model and prompt registries; manage feature and embedding pipelines; stand up serverless model endpoints with blue/green deploys; implement continuous evaluation (A/B), drift detection, usage telemetry, and human feedback loops; integrate with Purview/UC lineage and incident response.
• Demonstrate behaviors consistent with PEMCO's policies, values, code of ethics and business conduct.
• Authentically support the PEMCO Brand and constantly are on the lookout for top talent to join us to achieve our Mission to Worry Less and Live More.
• Other duties as assigned.

What You'll Bring:

• B.A./B.S. in Computer Science, Software/Systems Engineering, Data Engineering, Information Systems, or related field is required.
• Master's degree in a related field, such as Computer Science, Software Engineering, Artificial Intelligence, Machine Learning, Data Science or Data Engineering is preferred.
• 6-8+ years designing or operating cloud data platforms at enterprise scale.
• 3-4+ years hands on with Azure Databricks (or Databricks on cloud) including workspace administration, Unity Catalog, and Delta Lake.
• 3+ years building secure data pipelines (batch/streaming) with Spark, SQL, Python/Scala, and Delta; experience with Auto Loader and DLT Proven experience in Databricks workspace and Lakehouse architecture.
• 3+ years implementing CI/CD and infrastructure as code (Terraform strongly preferred) for data platforms
• Demonstrated success migrating or modernizing platforms (e.g., Hadoop/Synapse/SQL to Databricks Lakehouse) and establishing Dev/Test/Prod standards.
• Experience implementing lineage, cataloging, and governance (e.g., Purview/Unity Catalog) and integrating with enterprise SSO/SCIM.
• Experience in the insurance domain, particularly Auto, Home, and Umbrella.
• Databricks Certified Data Architecture Professional is preferred.
• Familiarity with agile software delivery methodologies such as Scrum is preferred.
• Familiarity with platforms and enabling tools such as Azure Machine Learning, Azure Databricks, Microsoft Fabric, Synapse Analytics, Power BI, Snowflake, and APIs like Azure OpenAI, Azure Cognitive Services, and Azure ML Endpoints is preferred.
• Databricks Certified Data Architecture Professional is preferred.
• Detail oriented with strong organizational, analytical, and communication skills.
• Service minded with experience deciphering ambiguous requests and taking ownership.
• Ability to create and maintain positive relationships with employees at all levels of the organization.
• Experience in the insurance industry and understanding of Auto, Home, and Umbrella insurance-related AI/ML applications.
• Strong knowledge of real-time data streaming frameworks like Apache Kafka, Azure Event Hubs, and Delta Live Tables for Databricks.
• Ability to work independently and manage tasks with minimal supervision.
• Experience with unsupervised learning techniques to identify patterns and anomalies in data.
• Strong knowledge of cloud-based security protocols, compliance standards, and governance frameworks is required.
• Proficiency with AI/ML frameworks and embedding AI solutions into data ecosystems. Familiarity with AI/ML tools like TensorFlow, PyTorch,, Azure ML is required.
• Proficiency in designing data models and architectures for both cloud and on-premises environments is required.
• Team Player: Is responsive, flexible, and able to succeed in a team-oriented, collaborative environment, building effective working relationships and partnerships with internal partners, customers, and vendors is required.
• Highly Analytical: Is passionate about working with disparate datasets bringing data together to answer business questions. Collaborates to create, manage, and translate data to meaningful insights. Can work with external vendors to integrate data for analysis; knows how to build efficient and scalable infrastructure and data models is required.
• Analytical and data-driven: Thinks analytically; a structured thinker who can put complex ideas into clear frameworks; uses data to conduct root cause analysis and develops high quality, consumable, and consistent metrics that drive strategic objectives and priorities is required.
• Problem Solver: Ability to analyze, diagnose and resolve complex unstructured problems quickly, efficiently, and collaboratively is required.
• Communicator: The ability to communicate clearly and informatively, verbally and in writing, with colleagues, customers, and the community in both technical and non-technical professional language is required.
• Job specific: Strong understanding of APIs, microservices architecture, and containerization (e.g., Kubernetes, Docker) is required.
• Job specific: Skilled in generative AI models like GPT, DALL·E, etc. is required.

What We Value:

In fostering a