Senior Data Engineer

Sr Data EngineerRole: Sr Data Engineer MoI: 2 Video Notes from Manager: Candidates must be located in San Antonio, TX, VA, DC, NY- will work onsite first week and then remotely, contract through end of year, ideally 5+ years of experience with Pyspark who has experience building a code framework using Pyspark and debugged using spark logs for performance tuning. The expected output is a highly performant code that will used as a backbone for the enterprise, able to handle complex business transformations and move data from Delta format (Databricks) to a data warehouse in Synapse Pyspark, Spark Architecture, Azure Databricks (Unity Catalog), Azure Synapse, Azure Data Factory (ADF), Azure Data Lake Storage (ADLS)ResponsibilitiesTakes a leading role in the design, development and implementation of processes to extract, transform and load data from disparate sources into a form that is consumable by analytics processes within a given project, reviewing deliverables to ensure high qualityEvaluates and resolves issues regarding data quality reviews, cleansing, data integration and migration, leveraging advanced technical knowledge and showing technical leadership in aspects of data engineering while driving continuous improvement effortsEnsures that the appropriate expectations, principles, structures, tools and responsibilities are in place to deliver the projectAnalyzes the latest industry trends such as cloud computing and distributed processing and infers potential impact on businesses (short and long-term)Provides advanced technical expertise to maximize efficiency, reliability and value from current data engineering processes and researches and monitors existing client base and industry developments to identify potential new product opportunities from emerging technologiesDevelops strong working relationships with peers across engineering, collaborating to develop leading data engineering solutionsDrives adherence to the relevant data engineering and data modelling processes, procedures, standards, and may input into the definition, maintenance and implementation of technology standardsQualificationsUses most of the related skills and technologies, and maintains an in-depth knowledge of the areaExperience designing and conducting research and experiments with recently developed technologiesHas successfully demonstrated domain of most skills and technologies during a large number of relevant projectsStrong experience in the use of complex technologies Computer ScienceTypical Education: Engineering, IT, Mathematics