Senior Databricks Platform Engineer

Job Description

Already have an IRS MBI? Let’s talk today.

\n

\n

Qualified candidates with an existing IRS MBI get priority interviews. We’re hiring across EDP platform engineering, pipeline development, cloud migration, Databricks, Informatica, AI/ML, data engineering, infrastructure, security/ATO, and networking.

\n

\n

Send me your résumé, and let’s connect.

\n

\n

Radiant | Remote

\n

Annual base salary: $225,000–$315,000 (Equivalent hourly rates for 1099 candidates)

\n

Citizenship: U.S. citizen or lawful permanent resident (green card holder)

\n

Background investigation: IRS Minimum Background Investigation (MBI) / Moderate-risk Public Trust

\n

\n

About Radiant

\n

Radiant is a fast-growing small business delivering digital services to public-sector clients. We specialize in end-to-end data engineering and AI-driven software development, helping government partners modernize critical systems and better serve their communities.

\n

\n

About the Role

\n

We’re hiring a Senior Databricks Platform Engineer to design, automate, and operate the Databricks Lakehouse Platform supporting IRS Enterprise Data Platform (EDP) Platform Engineering and Pipeline Development . As part of the Data Modernization & Governance team, you’ll build the secure, reliable foundation for data engineering, reporting, analytics, and AI/ML capabilities that support operational decision-making.

\n

\n

This is a platform ownership role and not solely a pipeline-development position. You’ll shape architecture, write production automation, implement governance controls, and improve performance, reliability, and cost efficiency. You’ll work closely with architecture, security, governance, and engineering teams to make the platform easier to operate and easier for others to use effectively.

\n

\n

What You’ll Do

\n

\n
• Build and operate the platform. Design, configure, and maintain Databricks environments on AWS, including workspaces, compute policies, clusters, SQL warehouses, and serverless capabilities where supported and authorized.
\n
• Make security and governance operational. Implement Unity Catalog catalogs, schemas, storage access, and least-privilege permissions for groups and service principals. Align identity integration, networking, encryption, auditing, and access controls with AO requirements.
\n
• Automate infrastructure and delivery. Use Terraform, Python, Databricks APIs, and Declarative Automation Bundles—formerly Databricks Asset Bundles—to provision resources and deploy workloads through version-controlled CI/CD pipelines with testing, environment promotion, and rollback procedures.
\n
• Develop reusable data-engineering patterns. Build and optimize ingestion frameworks, ETL/ELT pipelines, and orchestration using Apache Spark, Delta Lake, Lakeflow Spark Declarative Pipelines—formerly Delta Live Tables—and Lakeflow Jobs—formerly Databricks Workflows. Support integrations with enterprise data sources, BI platforms, and AI/ML environments.
\n
• Improve performance and cost efficiency. Diagnose Spark and SQL bottlenecks, test scalability, and optimize data layouts and compute configurations. Apply right-sizing, autoscaling, auto-termination, and workload isolation; track Databricks Unit (DBU) consumption and associated cloud costs to support forecasting and capacity planning.
\n
• Strengthen production reliability. Establish actionable monitoring, logging, alerts, and operational metrics. Troubleshoot incidents, lead root-cause analysis, manage runtime upgrades and releases, and implement and test recovery procedures. Support security audits and authorization-to-operate activities.
\n
• Help other engineers deliver effectively. Establish reusable templates, reference architectures, engineering standards, and practical runbooks. Mentor data engineers and platform users, explain technical trade-offs, and evaluate new capabilities against program needs.
\n

\n

\n

What You’ll Bring

\n

\n
• Hands-on production Databricks experience on AWS , including platform administration, Unity Catalog, compute management, SQL warehouses, and workflow orchestration—not just notebook development.
\n
• Strong Python, SQL, Apache Spark/PySpark, and Bash skills, with demonstrated experience troubleshooting distributed processing and improving workload performance.
\n
• Experience with Terraform, Git, and CI/CD tools , such as GitHub Actions, GitLab CI, or Azure DevOps, including automated testing, configuration management, and deployment across environments.
\n
• Practical knowledge of AWS security and networking , including IAM, S3 access, private connectivity, encryption, and enterprise identity integration.
\n
• Experience with Delta Lake, data ingestion, ETL/ELT, and lakehouse architectures , including operating and troubleshooting production data pipelines.
\n
• Experience using observability tools , such as Grafana, Datadog, or Amazon CloudWatch, to investigate incidents, identify performance trends, and guide capacity and cost decisions.
\n
• Strong engineering judgment, clear communication, and the ability to work independently while collaborating across architecture, security, governance, and delivery teams.
\n

\n

\n

Education and Experience

\n

Bachelor’s degree with 8+ years of IT experience , including 6+ years of relevant specialized experience .

\n

\n

Citizenship and Background Investigation

\n

Candidates must be U.S. citizens or lawful permanent residents (green card holders) . For moderate-risk IRS contractor positions, lawful permanent residents must have at least three consecutive years of U.S. residency from the date of legal entry as a lawful permanent resident , consistent with IRS requirements.

\n

\n

This position requires an IRS Minimum Background Investigation (MBI) and a favorable moderate-risk Public Trust suitability determination for access to sensitive Federal Tax Information (FTI). Candidates must satisfy and maintain the applicable IRS suitability and access requirements. Public Trust is not a national security clearance.

\n

\n

Helpful, but Not Required

\n

\n
• Experience supporting federal or other regulated environments, including FedRAMP-related requirements, security assessments, audit evidence, and authorization-to-operate processes.
\n
• Familiarity with Docker, Kubernetes, and microservices patterns , particularly for services and integrations surrounding the data platform.
\n
• Experience integrating Databricks with enterprise governance tools, BI platforms, or production AI/ML environments.
\n

\n

\n

Relevant credentials are welcome, including Databricks Certified Data Engineer Associate or Professional , the Databricks Data + AI Platform Administrator accreditation , AWS Certified Data Engineer – Associate , or AWS Certified Solutions Architect – Professional .

\n

\n

Don’t check every technical box? We encourage you to apply if you bring strong production Databricks experience and sound engineering judgment. You do not need experience with every example tool listed; the citizenship, background investigation, and education/experience requirements still apply.

\n

\n

Compensation and Benefits

\n

The anticipated annual base salary range is $205,000–$275,000 . Individual compensation will reflect relevant experience, qualifications, geographic location, and applicable contract requirements.

\n

\n

Benefits and additional compensation:

\n

\n
• Comprehensive health insurance
\n
• 401(K) plans with employer contributions
\n
• Professional development opportunities
\n
• A collaborative and inclusive work environment
\n

\n

\n

Interview Process

\n

Our process includes an introductory conversation, a technical discussion of Databricks architecture and production troubleshooting, and a final conversation with the team. We’ll focus on systems you have built or operated, the trade-offs you made, and the results you delivered.

\n

\n

How to Apply

\n

Submit your résumé through View email address on ziprecruiter.com . We’re particularly interested in the platforms you’ve operated, the automation you’ve built, and the performance, reliability, or cost improvements you’ve delivered.

\n

\n

Radiant is an equal opportunity employer. We welcome qualified applicants from all backgrounds and provide reasonable accommodations throughout the hiring process.