Azure Data Engineer

Omega Solutions

As a Data Engineer on the Data Science and Platform team, you will:

Develop and maintain data pipelines including solutions for data collection, management and usage.
Work closely with Data Scientists to optimize and reengineer model code to be modular, efficient and scalable, and to deploy models to production.
Maintain and enhance our data and computation platform.
Develop and implement solutions for data quality validation and continuous improvement.
Manage, execute and monitor weekly and monthly production operations; resolve and escalate production issues as appropriate.
Partner with business stakeholders to understand business and technical requirements, plan and execute projects, and communicate status, risks and issues.
Perform root cause analysis of system and data issues and develop solutions as required.

Qualifications:

BA/BS in business, computer science or other technical field or 5+ years of appropriate work experience
Proficient in Azure SQL Data Warehouse, T-SQL and database architecture.
Implement data warehouses and analysis services as part of a modern data warehouse (MDW) design
Experience working with cloud-based technologies, including relational databases, data warehouse, big data (Spark), orchestration/data pipeline tools, data frames concepts, Spark Streaming and Spark SQL
Experience with Azure Analytics stack, Azure Data Lake, Azure Databricks, Azure Data Factory.
Experience with ETL data pipelining concept.
Proficient in Power BI
Experience working in software engineering, and can demonstrate best practices for project management, quality control, and product development.
Proven track record of collaborative development in an agile team environment.
Experience with Python, Scala. Expertise in Pyspark or Scala is a must.
Hands on experience of writing notebooks in Databricks using python for complex data aggregations, transformations, schema operations.

Preferred:

Vacancy posted more than 2 months ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Azure Data Engineer. Be the first to apply!