Databricks Architect

Who We Are: At PEMCO, we’re all about people – our customers, our employees, and the community. We’re a mutual insurance company owned by our Northwest policyholders. We provide auto, home, renters, and boat coverage. Job Description At PEMCO, we’re all about people – our customers, our employees, and the community. We’re a mutual insurance company owned by our Northwest policyholders. We provide auto, home, renters, and boat coverage. Recognized by Forbes as one of America’s Best Insurance Companies in both Auto and Home for 2025, based on customer survey feedback, and by Newsweek as one of America’s Greatest Midsize Workplaces 2025. We are consistently recognized for our outstanding customer service, employee expertise, community partnerships, and social impact programs. All of which makes PEMCO a great place to work! Our social impact programs motivate high achievement by youth in education; build stronger and greener communities; and increase safety at home, on the road, and at play. We’re committed to diversity, equity, inclusion, and belonging, and to fostering an inspiring and inclusive workplace. These efforts create and cultivate an environment that builds fairness and understanding, encourages collaboration and flexibility, and celebrates all the ways in which we’re different and the same – enabling all individuals to achieve their full potential. Why We Need You PEMCO is seeking a Databricks Architect to join our community. The Databricks Architect leads the end-to-end architecture, governance, and scalability of the Azure Databricks Lakehouse platform at PEMCO. The Databricks Architect is responsible for landing zone design, security and compliance, data governance with Unity Catalog, performance and cost optimization, and cross environment (Dev/Test/Prod) standards. The architect partners closely with Enterprise Architecture, Security, Networking, Data Engineering, Analytics, and Product teams to deliver a secure, reliable, and cost efficient platform that accelerates use cases across underwriting, claims, pricing, customer engagement, fraud, and enterprise analytics. What You’ll Be Doing Own the Azure Databricks landing zone architecture: subscription and resource group strategy, workspace hierarchy, VNet injection, Private Link/endpoints, firewall rules, and cross region DR/BCP approaches. Define and implement Lakehouse patterns (bronze/silver/gold) using Delta Lake, including CDC ingestion with Auto Loader, streaming with Structured Streaming and Delta Live Tables (DLT), and table optimization (OPTIMIZE/Z ORDER/VACUUM policies). Establish and govern Unity Catalog: metastore design, external locations, schema/catalog standards, lineage integration, access models (RBAC/ABAC), and data protection (row/column level security, masking, tags). Create platform governance standards: SCIM provisioning, SSO/SAML, secret management (Key Vault backed scopes), cluster and pool policies, IP access lists, workspace and job ACLs, repo standards, and promotion workflows across Dev/Test/Prod. Build CI/CD and release engineering for Databricks (Repos, Git integration, ADO/GitHub Actions, Terraform/Bicep/ARM), including environment promotion, approvals, and automated quality gates. Implement observability and reliability practices: platform health dashboards, UC audit log analytics, job/runtime metrics, event logs, incident response runbooks, SLO/SLI reporting, and capacity planning. Drive performance engineering: benchmark Spark workloads, Photon adoption, caching strategies, file sizing and partitioning, concurrency controls, and SQL warehouse tuning. Lead FinOps for Databricks: budgets and tagging strategy, cluster policy guardrails (autoscale, instance families, spot usage, idle timeouts), serverless/SQL cost controls, and unit economics reporting. Partner with Security & Compliance to meet enterprise and regulatory requirements (encryption at rest/in transit, key management, PII handling, audits, retention policies, vulnerability management). Integrate the platform with Azure and enterprise services: ADLS Gen2, Event Hubs/Kafka, AAD, Purview, Key Vault, Log Analytics, Tableau, Salesforce, Genesys, and Delta Sharing. Enable MLOps and analytics at scale: MLflow model registry standards, feature engineering conventions, real time/batch inference patterns, Workflows orchestration, and blue/green or canary deployment strategies. Coach, guide, and review solution designs from data engineers and analysts; conduct architecture/design reviews and code reviews; publish reference architectures, standards, and reusable templates. Develop and maintain platform documentation, onboarding materials, and runbooks; lead knowledge transfer and internal training. Lead design and rollout of Databricks Mosaic AI (Foundation Model APIs, Model Serving, Vector Search, and Agent Framework) for enterprise GenAI use cases (RAG over governed data, copilots, and agents). Define prompt/response patterns, evaluation frameworks, guardrails (safety filters, PII redaction), offline/online tests, latency/error SLOs, and cost controls. Build LLMOps on the Lakehouse: instrument MLflow/Unity Catalog–backed model and prompt registries; manage feature and embedding pipelines; stand up serverless model endpoints with blue/green deploys; implement continuous evaluation (A/B), drift detection, usage telemetry, and human feedback loops; integrate with Purview/UC lineage and incident response Demonstrate behaviors consistent with PEMCO's policies, values, code of ethics and business conduct. Authentically support the PEMCO Brand and constantly are on the lookout for top talent to join us to achieve our Mission to Worry Less and Live More. Other duties as assigned. What You'll Bring B.A./B.S. in Computer Science, Software/Systems Engineering, Data Engineering, Information Systems, or related field is required. Master’s degree in a related field, such as Computer Science, Software Engineering, Artificial Intelligence, Machine Learning, Data Science or Data Engineering is preferred. 6-8+ years designing or operating cloud data platforms at enterprise scale. 3-4+ years hands on with Azure Databricks (or Databricks on cloud) including workspace administration, Unity Catalog, and Delta Lake. 3+ years building secure data pipelines (batch/streaming) with Spark, SQL, Python/Scala, and Delta; experience with Auto Loader and DLTProven experience in Databricks workspace and Lakehouse architecture. 3+ years implementing CI/CD and infrastructure as code (Terraform strongly preferred) for data platforms Demonstrated success migrating or modernizing platforms (e.g., Hadoop/Synapse/SQL to Databricks Lakehouse) and establishing Dev/Test/Prod standards. Experience implementing lineage, cataloging, and governance (e.g., Purview/Unity Catalog) and integrating with enterprise SSO/SCIM. Experience in the insurance domain, particularly Auto, Home, and Umbrella. Databricks Certified Data Architecture Professional is preferred. Familiarity with agile software delivery methodologies such as Scrum is preferred. Familiarity with platforms and enabling tools such as Azure Machine Learning, Azure Databricks, Microsoft Fabric, Synapse Analytics, Power BI, Snowflake, and APIs like Azure OpenAI, Azure Cognitive Services, and Azure ML Endpoints is preferred. Databricks Certified Data Architecture Professional is preferred. Detail oriented with strong organizational, analytical, and communication skills. Service minded with experience deciphering ambiguous requests and taking ownership. Ability to create and maintain positive relationships with employees at all levels of the organization. Experience in the insurance industry and understanding of Auto, Home, and Umbrella insurance-related AI/ML applications. Strong knowledge of real-time data streaming frameworks like Apache Kafka, Azure Event Hubs, and Delta Live Tables for Databricks. Ability to work independently and manage tasks with minimal supervision. Experience with unsupervised learning techniques to identify patterns and anomalies in data. Strong knowledge of cloud-based security protocols, compliance standards, and governance frameworks is required. Proficiency with AI/ML frameworks and embedding AI solutions into data ecosystems. Familiarity with AI/ML tools like TensorFlow, PyTorch, , Azure ML is required. Proficiency in designing data models and architectures for both cloud and on-premises environments is required. Team Player: Is responsive, flexible, and able to succeed in a team-oriented, collaborative environment, building effective working relationships and partnerships with internal partners, customers, and vendors is required. Highly Analytical: Is passionate about working with disparate datasets bringing data together to answer business questions. Collaborates to create, manage, and translate data to meaningful insights. Can work with external vendors to integrate data for analysis; knows how to build efficient and scalable infrastructure and data models is required. Analytical and data-driven: Thinks analytically; a structured thinker who can put complex ideas into clear frameworks; uses data to conduct root cause analysis and develops high quality, consumable, and consistent metrics that drive strategic objectives and priorities is required. Problem Solver: Ability to analyze, diagnose and resolve complex unstructured problems quickly, efficiently, and collaboratively is required. Communicator: The ability to communicate clearly and informatively, verbally and in writing, with colleagues, customers, and the community in both technical and non-technical professional language is required. Job specific: Strong understanding of APIs, microservices architecture, and containerization (e.g., Kubernetes, Docker) is required. Job specific: Skilled in generative AI models like GPT, DALL·E, etc. is required. What We Value In fostering a culture where everyone thrives, PEMCO embodies a set of core traits that guide our collective efforts. These traits include: Make Agile Decisions: Define desired outcomes. Use a consistent, inclusive approach. Drive decisions with a sense of urgency. Pivot decisions quickly as supported by data. Build Trust: Listen to understand and use direct and open communication. Engage in healthy debate with mutual respect. Find solutions and move forward together. Own Results: Commit to desired outcomes. Define and achieve success metrics. Anticipate and mitigate risks, and always have a plan B. Take accountability for and learn from outcomes. Empower Execution: Take action to drive desired outcomes. Act with initiative, mastery, and purpose. Explore Fearlessly: Make swift, iterative progress through testing and learning, create a safe place to fail fast and learn from mistakes, and continuously improve. Value People: Appreciate and care about unique experiences and perspectives Respect the dignity of others and have empathy for people Authentically and meaningfully support and recognize people in real time. Physical Demands And Working Conditions Physical Demands Office work: Office work involves working at a desk most of the time, using a stand‑up/sit‑down adjustable desk. Exerting up to 10 pounds of force occasionally and/or negligible amount of force frequently to lift, carry, push, pull, or otherwise move objects. Fingering, grasping, and repetitive motion typically is every day, and walking and standing are required occasionally. Working Conditions Requires occasional overnight travel (by car or air). Requires employee to be at work on-site during specified or core hours. Requires close visual acuity to perform an activity such as preparing and analyzing data and figures, viewing a computer screen, visual inspections, assembly of parts, or extensive reading. Requires ability to work a flexible schedule to meet business needs. Requires working indoors generally in environmentally controlled conditions. Job tasks may need to be performed in close physical proximity to other people. Compensation The pay range for this role is shown below. Compensation decisions are determined based on an individual’s qualifications, job‑related knowledge, skills, and experience. Greater Seattle area target pay range: $170,330–$208,181. The full pay range is $141,941–$236,569. Outside Greater Seattle area target pay range: $150,319–$183,723. The full pay range is $125,266–$208,777. Greater Seattle Area is defined as working within approximately 100 miles of Seattle. Outside Greater Seattle is defined as working approximately 100 miles or more from Seattle. Benefits Regular part‑time PEMCO employees working at least 24 hours per week and regular full‑time PEMCO employees are eligible to elect coverage under medical, dental, and vision plans for themselves and their eligible family members with generous employer premium cost shares. In addition, as a benefits‑eligible employee, you are: Covered by employer‑paid basic life and accidental death & dismemberment insurance policies as well as long‑ and short‑term disability benefit coverages. Eligible to participate in PEMCO’s 401(k) plan, which includes a generous employer match (2 for 1 on the first 6% employee pre‑tax and/or Roth deferral, up to federal maximums). PEMCO provides the following paid leave programs for benefits‑eligible employees in their first year of PEMCO employment: Vacation accrues at a rate of 10 days for new hires, and increases based on a schedule to a maximum annual accrual of 25 days based on tenure. Granted four (4) personal days immediately upon hire. Paid holidays for the eight (8) holidays observed by PEMCO throughout the calendar year. Granted up to ten (10) days of sick leave immediately upon hire (pro‑rated based on hire date and full‑time/part‑time status), which is approximately 28 hours more per year than the Washington state‑required accrual. In addition, PEMCO provides paid time off for bereavement, jury duty, and employee volunteering in the community. Other Miscellaneous Benefit Programs Offered By PEMCO Include Flexible Spending Accounts Education Assistance Program after one year of service Scholarship program for children of PEMCO employees after one year of service Employee Assistance Program Well‑being program Discretionary taxable gifts and gift cards And other Perks & Benefits including discounts on computer software and hardware, cell phone plans, and rental cars Other compensation depending on role, contributions, and performance may include Discretionary bonuses Tiered sales commissions and/or incentives (from 5–25% of employee’s monthly sales) Employee referral bonuses Shift differential pay Equal Employment Opportunity At PEMCO, we celebrate and support our differences. We know employing a team rich in diverse thoughts, experiences, and opinions allows our employees, our products, and our community to flourish. PEMCO is honored to be an equal opportunity workplace. We are dedicated to equal employment opportunities regardless of race, color, ancestry, religion, sex, national orientation, age, citizenship, marital status, disability, gender identity, sexual orientation, or veteran status. Accommodations At PEMCO, people are at the heart of our business. We’re committed to providing an inclusive hiring experience for all candidates. If you need reasonable accommodations to apply for a role or participate in an interview, please contact us by email with your name, the job reference number(s), your preferred contact method, and a brief description of the accommodation needed. Requests are considered on a case‑by‑case basis in accordance with applicable disability laws, including the ADAAA. Applicants Have Rights Under Federal Employment Laws Family and Medical Leave Act (FMLA) Equal Employment Opportunity (EEO) Employee Polygraph Protection Act (EPPA) #J-18808-Ljbffr PEMCO Insurance