Databrick Architect

Job Description

Role : Databrick Architect

Location : Tarrytown New York (Onsite)

Key Responsibilities & Skills

\n
• Lead enterprise-scale data platform modernization initiatives, driving migration from AWS EMR, Apache NiFi, and legacy ETL frameworks to the Databricks Lakehouse Platform.
\n
• Architect and implement scalable Lakehouse solutions using Databricks, Delta Lake, Unity Catalog, and Databricks Workflows.
\n
• Design and govern end-to-end data pipelines for batch, streaming, CDC, and real-time data integration workloads.
\n
• Provide architecture leadership for large-scale AWS-based data ecosystems leveraging S3, IAM, Redshift, Glue Catalog, Airflow, and Databricks.
\n
• Develop enterprise data architecture standards covering data modelling, metadata management, lineage, governance, security, and compliance.
\n
• Drive adoption of Databricks best practices including Delta Live Tables (DLT), Auto Loader, Unity Catalog, Serverless Compute, Lakehouse Federation, and advanced optimization techniques.
\n
• Lead replatforming and migration programmes involving PySpark applications, Airflow DAGs, EMR workloads, Redshift integrations, and NiFi pipelines.
\n
• Partner closely with business stakeholders, enterprise architects, product owners, and data science teams to define technology roadmaps and target architectures.
\n
• Establish governance frameworks using Unity Catalog, data quality controls, observability, monitoring, and operational excellence practices
\n
• Provide technical leadership, mentoring, architecture reviews, design governance, and solution sign-offs across multiple delivery teams.
\n
• Lead architecture workshops, executive presentations, solution assessments, and technology evaluations.
\n

Mandatory Technical Skills

\n
• Databricks Lakehouse Platform
\n
• Delta Lake, Delta Live Tables (DLT)
\n
• Unity Catalog
\n
• Databricks Workflows
\n
• Auto Loader
\n
• PySpark, Spark SQL, Python
\n
• AWS (S3, IAM, Glue, Redshift, EMR, Lambda)
\n
• Apache Airflow
\n
• CDC & Data Migration Frameworks
\n
• Data Governance & Security
\n
• CI/CD, GitHub, Jenkins
\n

JD:

\n
• Lead enterprise-scale Databricks Lakehouse Architecture design and implementation on AWS.
\n
• Drive large-scale data platform modernisation and cloud transformation initiatives.
\n
• Architect scalable Medallion Architecture (Bronze, Silver, Gold) data platforms.
\n
• Lead migration of legacy EMR, NiFi, Redshift, and ETL workloads to Databricks.
\n
• Design high-performance batch and real-time data processing solutions.
\n
• Build robust ingestion frameworks using Auto Loader, Delta Lake, and Structured Streaming.
\n
• Define enterprise data governance and security standards using Unity Catalog.
\n
• Architect metadata-driven and reusable PySpark-based ETL/ELT frameworks.
\n
• Establish best practices for performance tuning, scalability, and cost optimisation.
\n
• Design and implement data quality, lineage, and observability frameworks.
\n
• Drive adoption of CI/CD, DevOps, Infrastructure as Code, and automation practices.
\n
• Collaborate with business, analytics, and engineering teams to define target-state architectures.
\n
• Conduct architecture reviews and provide technical leadership across multiple projects.
\n
• Mentor architects and senior engineers on Databricks and AWS best practices.
\n
• Design secure and scalable solutions leveraging AWS services (S3, Glue, Athena, Lambda, IAM, Redshift).
\n
• Implement and govern enterprise-wide data access, compliance, and security controls.
\n
• Evaluate and adopt latest Databricks capabilities such as DLT, Lakehouse Federation, Serverless, and MLflow.
\n
• Enable AI/ML, Generative AI, and advanced analytics use cases on the Lakehouse platform.
\n
• Create architecture roadmaps, migration strategies, standards, and governance frameworks.
\n
• Act as the primary technical advisor for customer leadership on data strategy and platform evolution.
\n