Azure Databricks Engineer

Azure Databricks Engineer

We are looking for a skilled Azure Databricks Engineer to design, develop, and optimize scalable data pipelines and cloud-based analytics solutions using Microsoft Azure and Databricks technologies. The ideal candidate should have strong experience in big data processing, ETL/ELT workflows, PySpark, and Azure cloud services.

In this role, you will:

• Design, build, and maintain scalable ETL/ELT pipelines using Azure Databricks and Apache Spark;
• Develop and optimize PySpark/Spark SQL notebooks for large-scale data processing;
• Implement batch and real-time data ingestion pipelines using Azure Data Factory (ADF), Databricks Workflows, and Structured Streaming;
• Manage and optimize Delta Lake/Lakehouse architecture, including Bronze, Silver, and Gold layers;
• Integrate data from multiple sources including APIs, databases, cloud storage, and third-party systems;
• Work with Azure services such as ADLS Gen2, Azure Synapse Analytics, Azure SQL Database, and Azure DevOps;
• Monitor, troubleshoot, and improve pipeline performance, scalability, and cost optimization;
• Implement data governance, security, access controls, and compliance standards using Unity Catalog and RBAC;
• Collaborate with Data Engineers, Data Scientists, Analysts, and Product teams to deliver reliable data solutions;
• Support CI/CD implementation, automation, and infrastructure-as-code practices.

If you are:

• A bachelor's degree in Computer Science, Engineering, Information Technology, or a related field;
• 5+ years of experience in Data Engineering or Big Data development;
• Strong hands-on experience with Azure Databricks and Apache Spark;
• Proficient in PySpark, Spark SQL, and SQL;
• Experience with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS);
• Knowledge of Delta Lake, Lakehouse architecture, and data modeling concepts;
• Experience with distributed data processing and performance tuning;
• Familiar with CI/CD pipelines, Git, and Azure DevOps.

Nice to have:

• Experience with Kafka, dbt, Airflow, or Snowflake.
• Knowledge of Terraform or Infrastructure as Code (IaC).
• Exposure to MLflow, machine learning pipelines, or MLOps.

As an Opinov8r, you will have:

• A digital-first approach: great talent knows no borders! You can work from wherever you are — we hire and collaborate with professionals worldwide.
• A remote work model: balance your professional and personal life with our flexible working conditions, empowering you to deliver your best from anywhere.
• Exciting projects: dive into impactful projects across industries that challenge and spark creativity.
• Opportunities to boost your expertise: grow your career with continuous learning, development opportunities, and hands-on experience.
• Join the best team ever: collaborate with our diverse and cross-cultural team of passionate technologists and creative thinkers.

How's the hiring process going:

• Initial Interview: If your background fits the role, we'll invite you for an interview with a Talent Acquisition Specialist.
• Technical Interview: Depending on the position, you may complete a technical assessment or test task.
• Client Interview
• Final Decision: After all steps, we'll get back to you with the result and next steps.