Databricks Platform & Database Engineer

Databricks Platform & Database EngineerCoStar Group (NASDAQ: CSGP) is a leading global provider of commercial and residential real estate information, analytics, and online marketplaces. Included in the S&P 500 Index, CoStar Group is on a mission to digitize the world's real estate, empowering all people to discover properties, insights and connections that improve their businesses and lives.We have been living and breathing the world of real estate information and online marketplaces for over 35 years, giving us the perspective to create truly unique and valuable offerings to our customers. We've continually refined, transformed and perfected our approach to our business, creating a language that has become standard in our industry, for our customers, and even our competitors. We continue that effort today and are always working to improve and drive innovation. This is how we deliver for our customers, our employees, and investors. By equipping the brightest minds with the best resources available, we provide an invaluable edge in real estate.We are looking for a Databricks Platform & Database Engineer with hands-on experience in enterprise Databricks environments on AWS and traditional database engineering spanning relational and non-relational systems. In this role, you will serve as the primary owner of our Databricks environment on AWS responsible for administration, automation, governance, and platform reliability end to end. Alongside that, you will manage, optimize, automate, and govern a broader portfolio of relational databases (SQL Server, Aurora, PostgreSQL, MySQL) and nonrelational data stores across both AWS cloud and on-premises data centers.CoStar is investing heavily in artificial intelligence to accelerate software delivery, improve platform observability, automate research and business workflows, enhance product experiences through intelligent agents and skills, and improve data quality across enterprise systems. Successful candidates will demonstrate a willingness to learn, adopt, and apply AI technologies responsibly to solve complex business and engineering challenges.This position is located in Arlington, VA and is in office Monday through Thursday with work from home on Friday.ResponsibilitiesDeploy, administer, and maintain Databricks workspaces on AWS, including workspace configuration, network settings, access management, Runtime versions, cluster configurations, libraries, and init scripts across dev, staging, and productionManage Unity Catalog objects (catalogs, schemas, tables, external locations) and administer workspace security: users, groups, service principals, permissions, and enterprise identity provider integrationsLead platform upgrades, workspace migrations, and full environment lifecycle managementMonitor, optimize and right?sizing clusters, SQL warehouses, and job compute for performance and cost; analyze Spark executor logs and query plans to diagnose and resolve bottlenecksSupport data ingestion from relational and non-relational sources into Databricks via AWS, DMS or native Lakeflow ingestion servicesLead the design and implementation of end-to-end data workflows, covering the reliable distribution of curated data to downstream systems including SQL Server, DynamoDB, S3, and other enterprise data consumers.Design, implement, and administer relational and non-relational databases such as SQL Server, Aurora, PostgreSQL, MySQL, and DynamoDB across AWS (RDS/Aurora) and on-premises data centersManage provisioning, patching, backups, replication, failover, version upgrades, and cross-platform migrations between cloud and on-premises environmentsApply database security best practices including encryption at rest and in transit, access control, and auditingDevelop IaC solutions using Terraform for Databricks workspaces, AWS RDS, Aurora, and supporting resourcesAutomate operational tasks: cluster lifecycle, database provisioning, job scheduling, and backup management across cloud and on-premises environmentsBuild and maintain CI/CD workflows for platform configuration and pipeline codeImplement proactive monitoring and alerting across Databricks jobs, pipelines, clusters, SQL warehouses, and all database instancesImplement and enforce cluster policies, SQL warehouse governance, and workspace hardening.Manage Databricks audit logs, lineage, and compliance reportingSupport SLA-driven 24/7 production environments; lead root cause analysis, troubleshoot incidents, and author operational runbooksPartner with data scientists, business analysts, DBAs, and DevOps engineers to deliver platform implementations and respond to ad-hoc data needsBasic QualificationsBachelor's Degree required from an accredited, not for profit, in person, university or college.A track record of commitment to prior employers.Familiarity with large language models (LLMs), prompt engineering, or agent-based architectures.5+ years of experience in database engineering or data platform roles, across both relational/non-relational database administration and Databricks platform ownershipProven, hands-on experience administering enterprise Databricks environmentsCollaborate with cross-functional teams: Work closely with business analysts, data scientists, DBAs, and DevOps engineers to ensure successful data platform implementations.Ability to retrieve, synthesize, and present critical data in structures that is immediately useful to answering specific ad-hoc questionsKnowledge of data governance and compliance best practices.Deep expertise in Databricks platform administration: Unity Catalog, workspace security, cluster management, and Auto Loader on AWSStrong relational database engineering skills: schema design, query optimization, indexing, stored procedures, replication, and failover across SQL Server, Aurora, PostgreSQL, and/or MySQLAdvanced SQL and proficiency in PySpark and PythonInfrastructure-as-Code experience with Terraform; CI/CD pipeline managementExperience with cost optimization initiatives.Experience supporting 24/7 operational environmentsExperience with data governance frameworks, compliance practices, and cloud cost optimizationPreferred QualificationsWorking knowledge of Data Visualization tools like Looker and PowerBIWorking knowledge of non-relational databases: DynamoDB, MongoDB, or equivalent NoSQL platformsFamiliarity with AI governance frameworks, model lineage, auditability, and compliance requirements for ML systemsExperience with kafka or other real-time streaming platforms, Postgres/RDBMSExperience with Snowflake AdministrationExperience working with and managing large consumer datasets to derive insightsExperience with FinOps / cloud cost optimizationExperience implementing access controls, data privacy protections, and monitoring for AI workloads.Familiarity with ML tooling: MLflow, TensorFlow, PyTorch, or Scikit-learnUnderstanding of Lakehouse AI architecture and how ML pipelines integrate with Unity Catalog and Delta LakeWe welcome all qualified candidates who are currently eligible to work full-time in the United States to apply. However, please note that CoStar Group is not able to provide visa sponsorship for this position.The final salary or hourly rate offered for this role will fall within the range set forth below based on a variety of factors, including but not limited to, geographic location, skills, and competencies.Base Compensation: $136,000 - $177,000 annually.CoStar Group is an Equal Employment Opportunity Employer; we maintain a drug-free workplace and perform pre-employment substance abuse testing