Sr. Technical Lead

Sr. Technical Data Lead

Position Description: Seeking a Sr. Technical Data Lead who has experience developing and deploying data pipelines and applications on Microsoft Azure as well as exposure to AI/ML concepts. Common tools used are Python, PySpark, SQL, and Databricks. This individual should be able to lead teams and multiple workstreams of data engineers, data scientists, and data analysts as well as manage expectations with client stakeholders. The candidate will work jointly with the cloud architects to understand the overall vision of the solution and guide team members to execute on that vision. At times, this candidate will need to be able to jump in and contribute individually to the build out of the solution.

Your future duties and responsibilities:

• The Sr. Technical Data Lead will be responsible for development, deployment, maintenance and support of data pipelines on the MS Databricks platform
• Will streamline code, rationalize datasets and tables to arrive at single source of truth
• Will build detailed architecture diagrams or Entity Relationship Diagrams for all workflows and processes
• Will lead multiple data engineering workstreams towards a common vision
• Will manage stakeholder expectations and act as the liaison between architects and development team

Required qualifications to be successful in this role:

• 4 to 5 years of hands on experience with cloud automation, scripting in an Azure environment, and data transformation
• Understanding of high level AI/ML concepts and in particular experience with feature engineering and natural language processing
• Experience transforming unstructured data (jsons, text, etc.) for inputs in AI/ML models
• Experience in development of apache Spark code using PySpark or Scala
• Experience in using Azure Databricks Platform
• Experience in using Azure Data Factory to call Azure Databricks notebook activities
• Experience in using Azure DevOps based repositories
• Good fundamental knowledge about distributed computing, RDBMS and Dimensional modeling concepts
• Exposure to Azure Storage (Blob/Data Lake Store), Azure Catalog, Databricks Delta