Databricks Engineer / Architect

Position: Databricks Engineer / Architect
Employment Type: Contractual
Work Arrangement: Remote
Experience: 10+ years overall, with 4+ years of hands-on Databricks experience
Location: Remote
About the Role
We are seeking an experienced Databricks Engineer / Architect to design, develop, and optimize modern data platforms and large-scale data engineering solutions using Databricks and cloud technologies.
The ideal candidate will have strong hands-on expertise in Databricks, Apache Spark, Python/SQL, Delta Lake, data architecture, ETL/ELT pipelines, and cloud platforms. This role requires someone who can operate at both the engineering and architecture levels—translating business requirements into scalable, secure, and high-performance data solutions.
Key Responsibilities
Design and architect scalable, reliable, and high-performance data platforms using Databricks.
Develop and optimize data pipelines using PySpark, Python, SQL, Spark, and Delta Lake.
Design modern Lakehouse architectures and implement enterprise-grade data solutions.
Develop batch and streaming data pipelines and integrate data from multiple sources.
Implement Delta Lake capabilities including schema evolution, partitioning, optimization, and data lifecycle management.
Work with Databricks Workflows, Jobs, notebooks, clusters, and related platform capabilities.
Design and implement data ingestion, transformation, and orchestration frameworks.
Optimize Spark workloads, Databricks clusters, SQL queries, and data pipelines for performance and cost.
Establish and implement data engineering best practices, coding standards, CI/CD, and deployment processes.
Implement appropriate security, governance, access controls, and data quality mechanisms.
Collaborate with data scientists, analysts, application teams, cloud engineers, and business stakeholders.
Provide technical leadership and mentorship to data engineering teams.
Evaluate existing data architectures and recommend improvements, modernization strategies, and technology adoption.
Troubleshoot complex production issues and provide root-cause analysis and long-term solutions.
Participate in architecture reviews, technical design sessions, and documentation of enterprise data solutions.
Required Technical Skills
Databricks & Data Engineering
Strong hands-on experience with Databricks
Advanced knowledge of Apache Spark / PySpark
Strong proficiency in Python and SQL
Extensive experience with Delta Lake
Experience designing and implementing scalable ETL/ELT pipelines
Strong understanding of Lakehouse architecture and modern data platforms
Experience with batch and real-time/streaming data processing
Cloud Technologies
Strong experience with at least one major cloud platform:
Microsoft Azure - Azure Data Lake Storage, Azure Data Factory, Azure Synapse, Azure Key Vault, Azure DevOps
AWS - S3, Glue, Lambda, IAM, Redshift, CloudWatch
GCP - Cloud Storage, BigQuery, Dataflow, Pub/Sub, IAM
Preferred Qualifications
Databricks certifications such as Databricks Certified Data Engineer or Databricks Certified Data Engineer Professional
Experience with Unity Catalog and Databricks governance capabilities
Experience implementing enterprise Medallion Architecture (Bronze/Silver/Gold)
Experience with structured streaming and event-driven architectures
Knowledge of data modeling and dimensional modeling
Experience with Kafka or other messaging/streaming platforms
Experience with data quality frameworks and observability
Familiarity with MLOps and integration with machine learning platforms
Strong communication, documentation, and stakeholder-management skills
Education
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
Equivalent professional experience may be considered.
What We’re Looking For
The successful candidate should be:
A strong hands-on Databricks Engineer who can also think at the architecture level.
Comfortable working independently in a fully remote environment.
Capable of translating complex business requirements into scalable technical solutions.
Strong in problem-solving, troubleshooting, and performance optimization.
An effective communicator who can work with both technical and non-technical stakeholders.
Comfortable providing technical leadership and driving architecture decisions.