Data Engineer - Databricks, PySpark & Azure Data Factory

On-Site role

Job Description:

• The Data Engineer is responsible for designing, building, and supporting scalable data platforms, data products, application integrations, and analytics solutions that enable business decision-making across Power Delivery.
• This role develops and maintains enterprise data pipelines, integrates operational systems with cloud-based lakehouse platforms, and supports both low-code and pro-code application development using Microsoft Power Platform, OutSystems, and Databricks.
• The successful candidate will partner with business stakeholders, data analysts, application developers, and technology teams to deliver trusted, governed, and reusable data assets.

Key Responsibilities:

Data Engineering & Integration:

• Design, build, and support enterprise data pipelines using ETL and ELT methodologies.
• Develop scalable ingestion frameworks for structured, semi-structured, and streaming data.
• Build and maintain Databricks notebooks, workflows, and data pipelines.
• Ingest data from source systems using:
  • Azure Data Factory
  • Kafka
  • CDC technologies
  • APIs
  • Files and database sources
• Implement data quality checks, monitoring, and exception handling.
• Support Bronze, Silver, and Gold data architectures within the Databricks Lakehouse.

Databricks Development:

• Design and maintain Delta Lake tables and data products.
• Develop solutions using:
  • Databricks
  • PySpark
  • Spark SQL
  • Python
  • SQL
• Optimize lakehouse performance, scalability, and cost management.
• Support data sharing, governance, and catalog management using Unity Catalog.

Application Development:

• Build and enhance business applications using:
  • Microsoft Power Apps
  • Power Automate
  • Power Platform
  • OutSystems
• Develop integrations between Databricks and business applications.
• Partner with users to automate workflows and operational processes.

Data Governance & DevOps:

• Follow client governance and security standards.
• Implement CI/CD practices using GitHub and Azure DevOps.
• Maintain documentation, lineage, and metadata for enterprise data assets.
• Participate in architecture reviews and solution design activities.

Required Qualifications:

Bachelor's degree in:

• Computer Science
• Information Systems
• Data Analytics
• Engineering
• Related technical discipline

Experience:

• 3-7 years of experience in data engineering, analytics engineering, or software development.
• Experience building enterprise data pipelines and integrations.
• Experience with cloud-based data platforms.

Required Technical Skills:

Data Engineering:

• ETL / ELT design and development
• Data ingestion frameworks
• Data pipeline orchestration
• Data Warehousing concepts
• Data Lake and Lakehouse architectures
• Medallion architecture

Databricks:

• Databricks Workflows
• PySpark
• Spark SQL
• Delta Lake
• Unity Catalog
• Data Products

Cloud & Integration:

• Azure Data Factory
• APIs
• Kafka
• CDC technologies

Development:

• Python
• SQL
• GitHub
• Azure DevOps

Business Applications:

• Power Apps
• Power Automate
• Power Platform
• OutSystems

Preferred Qualifications:

• Utility industry experience.
• Power Delivery domain knowledge.
• Experience supporting Transmission or Distribution Analytics.
• Real-time data streaming experience.
• Knowledge of AI/ML enablement on Databricks.
• Experience with Power BI integration and Direct Query architectures.

About us:
At our organization, we take our mission and values to heart! We are on a mission to offer more and better jobs all over the world! Our goal is to care for you while you care for our clients and get you paid the highest pay possible. All our associates working with us are expected to embrace our RACE values: R - Results Matter, A- Approachable, C - Care, and E - Emergency i.e. work with a sense of urgency.

For more relevant job opportunities please visit our website:  Denken Solutions Careers