Senior Data Engineer

Sr Data Engineer

Role: Sr Data Engineer MoI: 2 Video Notes from Manager: Candidates must be located in San Antonio, TX, VA, DC, NY- will work onsite first week and then remotely, contract through end of year, ideally 5+ years of experience with Pyspark who has experience building a code framework using Pyspark and debugged using spark logs for performance tuning. The expected output is a highly performant code that will used as a backbone for the enterprise, able to handle complex business transformations and move data from Delta format (Databricks) to a data warehouse in Synapse Pyspark, Spark Architecture, Azure Databricks (Unity Catalog), Azure Synapse, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS)

Responsibilities

• Takes a leading role in the design, development and implementation of processes to extract, transform and load data from disparate sources into a form that is consumable by analytics processes within a given project, reviewing deliverables to ensure high quality
• Evaluates and resolves issues regarding data quality reviews, cleansing, data integration and migration, leveraging advanced technical knowledge and showing technical leadership in aspects of data engineering while driving continuous improvement efforts
• Ensures that the appropriate expectations, principles, structures, tools and responsibilities are in place to deliver the project
• Analyzes the latest industry trends such as cloud computing and distributed processing and infers potential impact on businesses (short and long-term)
• Provides advanced technical expertise to maximize efficiency, reliability and value from current data engineering processes and researches and monitors existing client base and industry developments to identify potential new product opportunities from emerging technologies
• Develops strong working relationships with peers across engineering, collaborating to develop leading data engineering solutions
• Drives adherence to the relevant data engineering and data modelling processes, procedures, standards, and may input into the definition, maintenance and implementation of technology standards

Qualifications

• Uses most of the related skills and technologies, and maintains an in-depth knowledge of the area
• Experience designing and conducting research and experiments with recently developed technologies
• Has successfully demonstrated domain of most skills and technologies during a large number of relevant projects
• Strong experience in the use of complex technologies Computer Science
• Typical Education: Engineering, IT, Mathematics