Azure Databricks Engineer

Azure Databricks Engineer

Location: Houston TX- Hybrid

Level: 5

Exp: 10+ Years

Core Responsibilities

Design and implement scalable data platforms and pipelines using Azure Databricks, Apache Spark, Pyspark, Delta Lake, and MLflow.

Design and implement data pipelines using Databricks, Spark, and Delta Lake for batch and streaming data.

Optimize Spark jobs for performance, scalability, and cost efficiency.

Develop and maintain Lakehouse architecture leveraging Databricks and cloud storage solutions.

Lead the migration from legacy platforms to Lakehouse architecture.

Develop batch and streaming data pipelines for ingestion, transformation, and analytics.

Establish standards for data governance, quality, and security.

Collaborate with stakeholders to align architecture with business goals.

Mentor data engineers and developers on Databricks best practices.

Integrate Databricks with tools like Power BI, Tableau, Kafka, Snowflake, and Azure Data Factory.

Required Skills & Experience

8–10+ years in data engineering or architecture roles.

8+ years of hands-on experience with Databricks and Azure.

Strong command of SQL, Python, Scala, and Spark.

Experience with CI/CD pipelines, DevOps, and orchestration tools like Airflow or Data Factory.

Familiarity with Azure cloud platforms.

Deep understanding of distributed computing, performance tuning, and data security.

Preferred Qualifications

Bachelor's or Master's in Computer Science, Data Science, or related field.

Certifications such as Azure Solution Architect, Azure Data Engineer, or Databricks Certified Data Engineer or Solutions Architect.

Experience with data mesh, data fabric, or enterprise data architectures.

Domain knowledge in finance, healthcare, or public sector is a plus.

Hadoop & Scala experience

Additional Insights from Internal Communications

Emphasis on DataOps, data orchestration, and data modeling.

Strong ERP knowledge (e.g., SAP, Salesforce) is valued.

Candidates should be hands-on, capable of self-exploration, and able to drive analytics adoption across business teams.

A structured questionnaire is often used to assess candidates on areas like Unity Catalog, Spark optimization, partitioning techniques, and data security.

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.