Data Engineer Databricks / Python / Database

Overview:
We are seeking an experienced Data Engineer with strong expertise in Databricks, Python, and Database engineering. The ideal candidate will design, build, and optimize scalable data pipelines, ensure data quality, and support advanced analytics and business intelligence initiatives across the organization.

• Design, build, and maintain ETL/ELT workflows on Databricks.
• Develop scalable data pipelines using Python and Spark.
• Optimize data transformation logic for performance and cost efficiency.
• Work with structured and unstructured data from various sources.
• Develop and manage data models, schemas, and tables across databases (SQL/NoSQL).
• Implement data quality, validation, and monitoring frameworks.
• Collaborate with Data Scientists, Analysts, and Cloud Engineers to support analytics and ML workloads.
• Ensure data security, governance, and compliance in all solutions.
• Troubleshoot pipeline failures and continuously improve pipeline reliability.
• Integrate data from APIs, streaming sources, and cloud storage.

Required Skills:
Core Technical Skills:-

• Databricks (mandatory) – Delta Lake, Notebooks, Spark SQL, Jobs, Workflows
• Python – Data processing, Spark APIs, automation, optimization
• Database Engineering – Strong SQL, query tuning, schema design, stored procedures
• Apache Spark – RDDs/DataFrames, optimisation, partitioning
• Cloud Platforms (one or more): AWS / Azure / GCP
• ETL/ELT tools: ADF, Glue, Dataflow, or equivalent
• Experience working with large-scale datasets
• CI/CD for data pipelines (Git, DevOps, Jenkins, etc.)