Principal Data Engineer

Principal Data Engineer (Path to Data Engineering Manager) Location: Remote - anywhere in the United States Employment Type: Full-Time Reports To: VP of Engineering Team: Will directly manage 3–5 offshore Data Engineers, with growth into a broader data engineering leadership role over time Springbrook Software is the cloud finance and ERP platform trusted by 3,000+ local government agencies across the United States. Our Cirrus platform delivers modern, multi-tenant SaaS solutions spanning finance, payroll, HR, utility billing, property tax, budgeting, permitting, and payments helps communities deliver the responsive, secure, and transparent services citizens expect. We have been named to the GovTech Top 100 for six consecutive years and backed by Five Arrows and Accel-KKR, we are aggressively expanding the Cirrus platform and investing deeply in AI to define the next generation of government technology. About the Role We're hiring a Principal Data Engineer who wants to stay deeply technical while growing into people leadership. This is fundamentally an individual-contributor / architect role and you'll be the primary hands-on builder of our data platform, writing production code in Databricks and Azure on a daily basis. Alongside that, you'll manage and mentor a small team of 3–5 offshore data engineers , giving you a natural runway toward a full-time Data Engineering Manager role as the team and platform scale. You'll be the technical owner of a global data platform that unifies data across 10+ products , supports both transactional and analytical workloads , and is architected from day one to onboard future acquisitions cleanly. You'll also design the platform's orchestration layer to support complex agentic AI workflows , including agent-to-agent (A2A) communication between autonomous systems. What You'll Do Platform Architecture & Hands-On Engineering Architect and build a unified, multi-tenant data platform on Databricks (Delta Lake, Unity Catalog, Workflows, MLflow) and Azure (ADLS Gen2, Azure Data Factory, Synapse, Azure Functions, Event Hub/Event Grid, Key Vault) that integrates data from 10+ product lines into a single source of truth. Design the platform to be growth-ready , with modular ingestion patterns, flexible schema onboarding, and governance structures that let new products or acquired companies plug in without re-architecture. Build and optimize pipelines that serve both transactional (OLTP-adjacent, low-latency) and analytical (OLAP, batch/streaming) use cases from a shared underlying platform, using the medallion architecture (bronze/silver/gold) and appropriate serving layers for each workload type. Design and implement the orchestration layer for agentic AI workflows , including agent-to-agent (A2A) communication , multi-step task automation, and event-driven coordination between autonomous agents operating on the data platform. Write and review production-grade PySpark, SQL, and Python code; this role stays in the codebase, not just in design docs. Own performance, scalability, and cost optimization across the Databricks/Azure environment. Team Leadership (Growing Over Time) Directly manage and mentor 3–5 offshore data engineers , setting technical direction, reviewing their code, and unblocking their work. Establish engineering standards, CI/CD practices, and testing/observability norms for the team to follow. As the platform and team grow, take on expanded management scope, with a clear path toward a Data Engineering Manager Cross-Functional & Strategic Partner with Product, Engineering, Security, and M&A/Corp Dev stakeholders to ensure the data platform's architecture supports business growth, including future acquisitions. Establish data governance, lineage, access control (RBAC/Unity Catalog), and compliance practices across a growing, multi-product data estate. Evaluate and integrate new tools (including emerging agentic AI orchestration frameworks) to keep the platform modern and extensible. What You'll Bring 8+ years of experience in data engineering or data architecture, with deep, hands-on You should be someone who still enjoys writing and reviewing code, not just reviewing architecture diagrams. Expert-level, hands-on experience with Databricks ( Delta Lake, Unity Catalog, Workflows/Jobs, cluster/job optimization, MLflow). Strong hands-on experience with the Azure data ecosystem (ADLS Gen2, Azure Data Factory, Azure Synapse Analytics, Azure Functions, Azure DevOps, Key Vault, Event Hub/Event Grid). Expert-level SQL and Python (PySpark) ; comfortable debugging Spark jobs at a deep level. Experience designing data platforms that serve both transactional and analytical workloads, and that can flexibly onboard new data sources (e.g., from M&A activity or new product lines). Familiarity with agentic AI orchestration concepts and frameworks (e.g., LangChain, LangGraph, AutoGen, or similar) and an interest in designing infrastructure for agent-to-agent (A2A) communication and multi-agent workflows. Prior experience leading or mentoring engineers (formal or informal) with genuine interest in growing into a people-management role over time. Experience working with offshore/distributed engineering teams, including async communication and code review across time zones. Solid understanding of data modeling, medallion architecture, CI/CD for data workflows (Azure DevOps/GitHub Actions), and Infrastructure-as-Code (Terraform/Bicep). Excellent communication skills; able to work directly with cross-functional stakeholders, including Corp Dev/M&A teams, on platform strategy. Nice to Have Databricks Certified Data Engineer Professional or Azure Data Engineer Associate/Expert certification. Experience integrating a data platform after a merger or acquisition or building with future M&A integration in mind. Experience with vector databases (FAISS, Pinecone, Weaviate) and RAG pipelines, in support of agentic AI use cases. Experience with Kafka, Event Hubs, or other streaming technologies for real-time transactional and analytical pipelines. Background in a regulated industry (healthcare, finance) with associated compliance requirements (HIPAA, SOC 2). Managing a Distributed Data Engineering Team You'll be responsible for managing 3–5 offshore data engineers. To give a concrete sense of the caliber and skill profile of the engineers you'd be leading, the team typically brings hands-on experience across: Big Data & Processing: Apache Spark (PySpark), Databricks (Delta Lake), Azure Data Factory, Azure Synapse, Kafka, Event Hub/Event Grid Cloud Platforms: Azure (ADLS, AKS, Azure Functions, Cosmos DB, Azure SQL, API Management), with some AWS/GCP exposure AI/ML & GenAI Tooling: Azure Machine Learning, Azure OpenAI Service, MLflow, LangChain, RAG pipelines, vector search DevOps/MLOps: Azure DevOps, GitHub Actions, Docker, Kubernetes (AKS), Terraform/Bicep, MLflow, Airflow Languages & Frameworks: Python (PySpark, FastAPI, Django/Flask), SQL, Java/Spring Boot, Node.js Monitoring & Observability: Azure Monitor, Application Insights, ELK Stack, Prometheus/Grafana Security & Compliance: Azure AD B2C, Key Vault, OAuth2/JWT, RBAC, HIPAA-ready configurations You'll set the architectural direction and coding standards this team executes against - reviewing their Databricks/PySpark work, guiding their approach to pipeline design, and developing their skills toward platform ownership over time. What We Offer Fully remote work, based anywhere in the U.S. A high-ownership architect role with a clear, structured path to engineering management. Competitive salary + equity/bonus (commensurate with experience and location). Comprehensive health, dental, and vision coverage. 401(k) with company match. Flexible PTO. Home office / equipment stipend. Budget for conferences, certifications, and continued learning. Applicants must have the unrestricted ability to work in the United States (sponsorship will not be offered) Springbrook Software is an Equal Opportunity Employer. Springbrook does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need