Databricks Senior Architect

Databricks Senior Architect

As a Databricks Senior Architect, you will lead the architecture and technical delivery of Primoris' Databricks-based data platform and use cases across data engineering, analytics, and AI/ML. You will step into an in-flight program, rapidly absorb context, and provide the architectural backbone needed to reliably deliver value from Primoris' data. You will act as a trusted technical advisor and delivery owner, working closely with Primoris' data, IT, and business teams (and implementation partners) to establish a clear architectural vision, define and prioritize use cases, and guide them into production on the Databricks Lakehouse. While this role is primarily focused on architecture, design, and technical leadership, you will also contribute hands-on where needed to accelerate critical workloads and mentor the wider engineering team.

This role is ideal for someone who wants to grow as a platform and solution leader, enjoys solving complex data and AI problems, and can connect technical decisions to measurable business outcomes (e.g., project profitability, safety and quality insights, forecasting, and operational efficiency).

Key Responsibilities

Platform & Architecture

• Define and maintain the end-to-end architecture for Primoris' Databricks Lakehouse, including Medallion layer patterns, ingestion, transformation, serving, and AI/ML/GenAI workloads.
• Design and implement scalable, secure Databricks patterns (including serverless where appropriate) that power batch, streaming, ML, AI Gateway and GenAI workloads, optimizing for performance, cost, security, and reliability.
• Architect and implement solutions using the Databricks Medallion architecture, Unity Catalog, and Delta Lake to support both analytical and operational use cases.
• Partner with Primoris' enterprise/solution architecture, security, infrastructure, and application teams to align Databricks with broader platform and integration standards.

Delivery & Operations

• Assist in architecting, developing, deploying, and migrating priority use cases and workloads into production on the Databricks Lakehouse.
• Serve as the first point of contact for technical challenges or questions related to development and production workloads (triage, root cause analysis, coordination of escalation as needed).
• Provide technical leadership and delivery oversight across hybrid teams, including internal employees, consulting partners, and offshore resources, ensuring seamless coordination, clear accountability, and consistent outcomes.
• Establish and champion engineering standards (CI/CD, automated testing, observability, monitoring/alerting, incident runbooks) to ensure production-grade delivery and operations.
• Design and review workflows, jobs, and streaming pipelines (e.g., Auto Loader, DLT, structured streaming) for robustness, recoverability, and SLA adherence.
• Implement Databricks product innovations, private previews, and upgrades following change management and release best practices, in coordination with Primoris IT and partners.

Data Engineering

• Build and review multi-language notebooks (SQL, Python/PySpark) with a primary focus on data engineering and data science/AI/ML workflows.
• Design and guide implementation of streaming and batch data pipelines (Auto Loader, DLT, Structured Streaming) handling structured, semi-structured, and unstructured data.

Governance, Security & Compliance

• Implement and oversee Row-Level Security (RLS), column masking, data encryption/decryption, and other data protection controls using Unity Catalog and cloud-native capabilities.
• Architect and govern enterprise GenAI platforms on Databricks, including Unity Catalog, AI Gateway, Model Serving, and Vector Search, ensuring secure, compliant, observable, and cost-effective AI solutions across the organization.
• Work with security and compliance stakeholders to define and implement data governance models, access patterns, data classification, auditing, and regulatory/contractual controls as required.
• Ensure all architectures and implementations align with Primoris' security, networking, and compliance standards for Azure and Databricks.

Stakeholder Management & Communication

• Participate in discovery workshops with business and technical stakeholders to understand current-state architecture, data flows, pain points, and priority use cases.
• Translate findings into strategic roadmaps, architecture options, and phased delivery plans with clear trade-offs, milestones, and ownership.
• Provide regular internal and external updates on progress, risks, and blockers to leadership, with clear options, mitigations, and recommendations.

Coaching, Standards & Reuse

• Define and evangelize pragmatic engineering and platform standards (coding patterns, data modeling, observability, cost optimization, workspace standards).
• Mentor engineers through design and code reviews, knowledge-sharing sessions, and pairing on complex tasks.
• Develop and maintain reusable assets (reference architectures, templates, starter notebooks, Terraform/IaC patterns, runbooks) to accelerate adoption and improve consistency.
• Demonstrate proactive ownership: surface risks early, drive decisions, and remove blockers to maintain delivery momentum and stakeholder confidence.

Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, Data Science, or equivalent practical experience.
• 8+ years of hands-on experience in Data, Analytics, or AI roles (e.g., data engineering, data architecture, analytics platform engineering), with 4+ years accountable for technical delivery and architecture/tech-lead responsibilities.
• Proven experience architecting and implementing Databricks solutions using the Medallion architecture, including ingestion, transformation, serving, and AI/ML workloads.
• Hands-on expertise implementing Databricks on Microsoft Azure (required), including familiarity with Azure security, networking, identity, and storage concepts.
• Experience designing and developing large-scale distributed systems and big data solutions using technologies such as Apache Spark™, Hadoop, or Cassandra.
• Proficiency in Python (including PySpark) and SQL, including performance tuning and best practices for large-scale data processing.
• Proficiency in designing and implementing Auto Loader, Delta Live Tables (DLT), and streaming solutions in Databricks to handle structured, semi-structured, and unstructured data.
• Experience with Unity Catalog and related capabilities (data governance, RBAC, lineage, auditing, and secure data sharing) or similar enterprise data governance tools.
• Demonstrated ability to attribute business value and outcomes to specific project deliverables and technical KPIs.
• Experience conducting discovery workshops, creating strategic roadmaps, performing business/process analysis, and managing or influencing the delivery of complex data/AI programs or projects.
• Expertise implementing end-to-end data security controls in Databricks (RLS, column masking, encryption, key management), in alignment with enterprise security policies.
• Evidence of continuous learning and staying current with the fast-evolving Databricks Lakehouse platform, Azure services, and modern data/AI patterns.
• Nice to have:
• One or more Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Data Architect Associate/Professional).
• Deep hands-on knowledge of Big Data Engineering or cloud DWH technologies (e.g., Synapse, Snowflake, BigQuery, Redshift).

Skills and Competencies

• Technical leadership: Ability to set architectural direction, make trade-off decisions, and guide engineers and partners toward production-ready solutions.
• Stakeholder management: Ability to manage expectations, influence decisions, and build trusted relationships with both technical and non-technical stakeholders (engineering, IT, security, operations, finance, business leadership).
• Communication: Excellent written and verbal communication skills; able to explain complex technical topics in clear, outcome-oriented language.
• Organization & execution: Exceptional organizational and time management skills; able to juggle multiple workstreams and drive them to completion.
• Attention to detail & quality: High standards for solution robustness, reliability, security, and documentation.
• Team building & mentoring: Strong leadership and team-building capabilities; experience mentoring engineers, establishing best practices, and raising the technical bar.
• Adaptability: Ability to adapt to changing priorities, work under pressure, and thrive in a fast-paced, evolving environment.
• Ownership mindset: Bias toward action, proactive identification of issues, and a strong sense of accountability for outcomes rather than just deliverables.