Azure Data Engineer

Azure Data Engineer

As a Data Engineer on the Data Science and Platform team, you will:

• Develop and maintain data pipelines including solutions for data collection, management and usage.
• Work closely with Data Scientists to optimize and reengineer model code to be modular, efficient and scalable, and to deploy models to production.
• Maintain and enhance our data and computation platform.
• Develop and implement solutions for data quality validation and continuous improvement.
• Manage, execute and monitor weekly and monthly production operations; resolve and escalate production issues as appropriate.
• Partner with business stakeholders to understand business and technical requirements, plan and execute projects, and communicate status, risks and issues.
• Perform root cause analysis of system and data issues and develop solutions as required.

Qualifications:

• BA/BS in business, computer science or other technical field or 5+ years of appropriate work experience
• Proficient in Azure SQL Data Warehouse, T-SQL and database architecture.
• Implement data warehouses and analysis services as part of a modern data warehouse (MDW) design
• Experience working with cloud-based technologies, including relational databases, data warehouse, big data (Spark), orchestration/data pipeline tools, data frames concepts, Spark Streaming and Spark SQL
• Experience with Azure Analytics stack, Azure Data Lake, Azure Databricks, Azure Data Factory.
• Experience with ETL data pipelining concept.
• Proficient in Power BI
• Experience working in software engineering, and can demonstrate best practices for project management, quality control, and product development.
• Proven track record of collaborative development in an agile team environment.
• Experience with Python, Scala. Expertise in Pyspark or Scala is a must.
• Hands on experience of writing notebooks in Databricks using python for complex data aggregations, transformations, schema operations.

Preferred:

• Experience with Cosmos DB
• Significant experience with utilizing Azure Analysis Services
• Experience with Airflow
• Elastic Search
• Experience in Java
• Experience building APIs and services using REST