Director of Data Engineering & Platform

Director of Data Engineering & PlatformCooley is seeking a Director of Data Engineering & Platform to join the Data team within the Innovation department.Position summary: As a leading technology law firm, Cooley is determined to become a leader in the digital practice of law. The Director of Data Engineering & Platform is a leadership role responsible for building, leading, and operating the Data Engineering & Platform function across Data Engineering and Platform & Operations. This role joins the Northstar Lakehouse program at a critical build phase, inheriting a live platform that is actively moving data into the Silver layer, with core Data Vault 2.0 structures being established across the firm's primary legal sector entities. The Director will develop high-performing Data Engineering and Platform Operations teams that deliver the Northstar Lakehouse from Silver layer foundations to gold layer products serving the firm’s AI and analytics ambitions. The platform is built on Databricks on AWS with a medallion architecture, Data Vault 2.0 methodology for non-financial data domains, and a two-track gold layer serving both legal intelligence and financial reporting consumption patterns. This is a hands-on leadership role that will build a greenfield data platform moving quickly, building correctly, and delivering at the pace a legal AI program demands. Specific duties and responsibilities include, but are not limited to, the following:Position responsibilities:Data Engineering & Platform Leadership:Lead and develop the Innovation Data Engineering & Platform function across Data Engineering and Platform & Operations. Build a high-performing, accountable team through clear expectations, consistent feedbackProvide direct leadership to the team and as the function matures and team members develop, make deliberate decisions regarding introducing a management layer beneath the DirectorDirect contract resources including transitional Enterprise Solution Architects and Contractor program resources, setting clear work agendas, managing delivery accountability, and making recommendations on resource extension or transition based on platform needs and team capability developmentPartner with the Director of Data on function strategy, hiring sequencing, budget, and the long-term capability roadmap for the Data Engineering & Platform function as the Northstar Lakehouse transitions from build phase to operational maturityDevelop the Senior Data Engineering Manager as the technical engineering leader this function needs over time. Provide direct, frequent feedback on delivery quality, team management, and technical decision-makingOwn workforce planning for the function, including permanent hire sequencing, contractor-to-permanent conversion decisions, and the right-sizing of the team as the platform moves from construction to steady-state operationsServe as direct supervisor and mentor to direct reportsProvide day-to-day supervision of direct reports, ensure compliance with assigned work hours and monitor for compliance with all firm and department policies. Manage staffing coverage, review and process time logs/time off requestsSupport business professional development and continued educational opportunitiesIn collaboration with immediate supervisor and central HR, participate in hiring, performance appraisals, counseling, termination and other employee lifecycle eventsNorthstar Lakehouse Build & Technical Delivery:Own the technical delivery of the Northstar Lakehouse build across the Bronze, Silver, and Gold layers of the Databricks medallion architecture. Drive reliable, high-quality data pipeline delivery across Data Vault 2.0 structures for non-financial legal sector domains and 3NF relational structures for the 3E financial and billing data domainEstablish and maintain the two-track gold layer architecture: a Data Vault Business Vault consumption layer serving legal intelligence, client and matter analytics, competitive intelligence, and AI-powered legal products, and a 3NF relational reporting layer serving financial reporting, billing analytics, and operational dashboards from the 3E legal billing and matter management systemEnsure the Data Engineering team implements Data Vault 2.0 pipelines in alignment with the methodology standards defined by the Principal Data Architect in the Platform Governance & Quality function. Create the conditions for the Principal Data Architect to provide effective methodology oversight, code review, and standards enforcement without frictionDrive the Bronze layer ingestion quality baseline, ensuring source-to-platform data completeness, reconciliation back to source systems, and structural validation are established across all active data sources before Silver layer certification is approvedOwn the delivery cadence of the Data Engineering function within the squad-based EPIC delivery model. Ensure Agile ceremonies are effective, sprint commitments are realistic and met, and the engineering team is unblocked and productive across active Data Domain SquadsPartner with the Director of Data Platform Governance & Quality on the certification workflow for Bronze and Silver layer data assets, ensuring data quality gates, governance controls, and access policies are embedded in the engineering build process rather than applied as post-delivery auditsPlatform Architecture & Operations:Own the Databricks platform architecture decisions for the Northstar Lakehouse, including Unity Catalog metastore configuration, compute tier design, job cluster and SQL warehouse architecture, Delta Lake table property standards, workspace organization, and platform security configurationDirect the Platform Architect in maintaining and evolving the Databricks platform as capabilities change, new Databricks features become available, and the platform's consumption patterns grow. Ensure the platform architecture stays current, performant, and aligned with the firm's AI and analytics ambitionsOwn the CI/CD framework for data pipeline deployment, ensuring reliable and automated promotion of pipeline code through development, UAT, and production environments. Hold the DataOps Engineer accountable for CI/CD framework operation and evolution as the pipeline estate scalesOversee Platform & Operations reliability across the live Databricks environment, including compute health, pipeline execution monitoring, incident management, and escalation protocols across a he platform that is live and serving active data consumers. Operational reliability is a must from Day 1Manage the integration of external tools and data sources with the Databricks platform, including Qlik Replicate for CDC ingestion, gateway and VM infrastructure coordination, and any additional tooling integrations required as the platform scalesTechnology Infrastructure Relationship & Cross-Functional Collaboration:Own the relationship with the firm's Technology function as the senior accountability point for all infrastructure dependencies that sit outside the Databricks platform. This includes network connectivity, gateway configuration, VM management, data center and cloud infrastructure coordination, and security policy alignment between the platform and the broader technical environmentWork closely with the Technology function on infrastructure dependencies for the Data Engineering & Platform team. Proactively surface infrastructure requirements early, build credible relationships with Technology counterparts, and resolve cross-functional blockers at the appropriate level without escalating unnecessarilyPartner with the Director of Data Platform Governance & Quality on cross-functional platform standards including data certification workflows, Unity Catalog governance configuration, access control policies, and the technical governance standards the Platform Governance & Quality function defines and the Engineering team implementsCollaborate with the Director of Data Products and the Intelligence & Client Services function on gold layer data product delivery, ensuring the engineering team understands the analytical and AI consumption requirements that drive gold layer design decisions and delivers the data structures those products depend onEngage with the Data Council through the Data Platform Governance & Quality function, providing engineering perspective on platform capability, delivery timelines, and technical feasibility of governance-driven data requirements surfaced by Data Domain Squads and working groupsAll other duties as assigned or requiredRequired:After orientation at Cooley LLP, exhibit proficiency in the Microsoft Office suite, iManage, and other firm applicationsAbility to work extended and/or weekend hours, as requiredAbility to travel, as required10+ years of experience in data engineering, data platform architecture, or enterprise solution architecture with demonstrated leadership of a data engineering function in a cloud-native environment with 5+ years of management experience in relevant rolesProven experience building a greenfield cloud data platform from the ground up, including hands-on involvement in medallion architecture design, Data Vault 2.0 implementation, pipeline framework establishment, and gold layer consumption pattern developmentDeep Databricks expertise including Unity Catalog configuration and governance, Delta Lake table design and optimization, Databricks SQL and job cluster architecture, compute tier strategy, workspace organization, and Databricks platform administration at an enterprise scaleStrong Data Vault 2.0 literacy, sufficient to assess whether Raw Vault hub, link, and satellite implementations are methodologically correctExperience designing and delivering a two-track or multi-layer gold layer architecture serving both analytical BI consumption and AI-powered product consumption from the same Silver layer foundationsDemonstrated experience directing ESA, contract, and transitional resources alongside a permanent engineering team, setting clear work agendas, managing delivery accountability, and making informed decisions about resource retention and transitionProven track record of effective leadershipExperience owning cross-functional infrastructure relationships, specifically coordinating between a data platform engineering team and an IT infrastructure or enterprise technology function on network, gateway, VM, and cloud connectivity dependenciesProven Agile delivery leadership in a squad-based data platform program, including sprint planning, EPIC delivery management, and definition of done discipline across multiple parallel data domain workstreamsComfort using GitHub for version control, CI/CD pipeline oversight, and engineering standards governance in a collaborative data platform environmentBachelor’s degreePreferred:Legal sector, professional services, or financial services experience with understanding of complex operational data environments including matter management, client relationship data, timekeeper and billing data, or comparable professional services data complexityExperience with 3NF relational modeling for operational or ERP systems, specifically in the context of building a financial or operational reporting gold layer alongside a Data Vault analytical layerExperience with dbt for transformation layer implementation including model development, testing frameworks, documentation standards, and the governance of dbt as a platform-wide transformation standardFamiliarity with Qlik Replicate, AWS DMS, or comparable CDC ingestion tooling for real-time or near-real-time data replication from operational source systems into a cloud data platformExperience with Informatica MDM, DQE, or comparable enterprise master data and data quality platforms and how they integrate with a Databricks lakehouse environmentFamiliarity with infrastructure-as-code tooling such as Terraform, AWS CloudFormation, or comparable for Databricks workspace and resource managementAI-assisted development fluency including Claude Code, GitHub Copilot, or comparable tools used to accelerate engineering delivery, code review, and documentation workflows across the teamExperience designing or operating a data platform that serves as the foundation for RAG pipelines, LLM fine-tuning datasets, vector search indexes, or comparable AI infrastructureCompetencies:Strong platform builder instincts with a genuine understanding of what it takes to move a greenfield data platform from foundations to gold layer products that business stakeholders can trust and consumeExcellent people leadership skills with a demonstrated ability to develop engineering talent, hold a team accountable to high standards, and make honest promotion and performance decisions without avoidanceDeep technical credibility in Databricks and cloud data platform architecture, sufficient to make real-time architectural decisions, assess engineering team work quality, and direct experienced ESA resources effectivelyStrong collaborative instincts with the ability to build productive working relationships across governance, data products, intelligence, and IT infrastructure functions without creating friction or competing for authorityExcellent judgment about when to be hands-on versus when to delegate, understanding that a greenfield program in build phase sometimes requires the Director to roll up their sleeves alongside the teamStrong delivery orientation with a track record of meeting commitments, running effective Agile ceremonies, and driving a data engineering team to consistent, high-quality output under program pressureExcellent written and verbal communication skills with the ability to represent the Data Engineering & Platform function credibly to executive stakeholders, the Technology function, and cross-functional platform partnersGrowth-oriented talent developer with genuine investment in the Senior Data Engineering Manager's career trajectory and the broader team's technical capability developmentStrong attention to detailStrong judgmentHigh level of professionalism at all timesDemonstrated ability to lead through influence and develop talent [if applicable]Unwavering ability to handle and maintain confidentiality regarding firm information, projects, and client dataCooley offers a competitive compensation and excellent benefits package and is committed to fair and equitable employment practices. EOE.The expected annual pay range for this position with a full-time schedule is $260,000 - $320,000.Please note that final offer amount will be dependent on geographic location, applicable experience and skillset of the candidate. We offer a full range of elective benefits including medical, health savings account (with applicable medical plan), dental, vision, health and/or dependent care flexible spending accounts, pre-tax commuter benefits, life insurance, AD&D, long-term care coverage, backup care for children and/or adults and other parental support benefits. In addition to elective benefit options, benefited employees receive firm-paid life insurance, AD&D, LTD, short term medical benefits as well as 25 days of Paid Time Off (“PTO”) and 10 paid holidays each year. We provide generous parental leave and fertility benefits. New employees will attend a detailed benefit orientation to learn more about our many benefits and resources.SummaryLocation: San Francisco; New York; Santa Monica; Los Angeles; Palo Alto; Chicago; Washington DC; Boston; Colorado; Seattle; San Diego; RestonType: Full time