Cloud Data Engineer

Data Engineer

This position is for a Data Engineer working on different cloud data services mainly Azure & AWS, Python, SSIS and to support data pipeline development & migration efforts along with PowerBI dashboard/report development. This includes development and application maintenance efforts. The individual would be a self-starter and capable of working with complex SQL queries, debugging Python and SQL routines, packages and code modification as needed.

Job Duties | Essential Functions

• Design, develop, document & maintain complex data pipelines using AWS data services such as EMR, Step Function, SNS, Event Scheduler to source data from multiple sources including RDBMS, Parquet files (on AWS S3 bucket), Excel, and text files.
• Design, develop, document & maintain complex data pipelines using Azure data services such as Logic App, ADF, ADLS gen 2, Azure Databricks, Azure Synapse workspace, Event hub, service bus to source data from multiple sources including RDBMS, Parquet files (ADLS gen 2, S3 bucket, GCP, on prem), Excel, and text files.
• Deploy & automate (using CICD pipeline) Azure, AWS cloud services using IaaC (Infra as a Code)
• Develop & design web solution using ReactJS, Redux & NodeJS
• Evaluate, debug, and modify existing complex Python/PySpark code, & SSIS packages in accordance with business requirements.
• Perform data analysis and test/debug software solutions
• Analyze existing ETL jobs, develop ELT pipelines & data warehouse applications or work to formulate logic for moderately complex new systems and devise moderately complex algorithms.
• Design and extract data from different data sources such as Guidewire© application or other on prem databases
• Apply effective programming security practices.
• Practices current development methods/techniques including CI/CD pipelines using AWS CodeCommit & AWS CodePipeline, Azure DevOps and establishes development standards (which include coding standards, documentation standards, and testing standards) to ensure the quality and maintainability of automated solutions.
• Familiarity with designing & developing data pipelines to source data from AWS/Azure cloud or on prem to Snowflake platform
• Following established change management code migration processes.

Skills And Abilities

• Experience in data ingestion, data transformation, and data loading
• Experience in Azure Data platforms such as Azure Databricks, Azure Synapse, Azure Data Factory, ADLS Gen2, etc (REQUIRED)
• Experience in AWS data platform such as AWS EMR, Step Function, Event Scheduler, SNS, S3, CodeCommit, CodePipeline (REQUIRED)
• Experience in Snowflake platform
• Working knowledge of Python, PySpark or equivalent
• Demonstrated broad problem-solving skills, including experience identifying and using code libraries and open-source forums
• Good written and verbal communication skills

Prior Work Experience

• 4 + Years of Data ingestion and transformation experience using AWS data services such as EMR, step function, event scheduler, SNS, Python & SQL Server SSIS (any combination)
• 2+ Years experience in Snowflake platform
• 8+ Years ETL experience with designing star/snowflake schema
• 2+ Years of experience with Azure Data platforms (ADF, Synapse workspace, Databricks, Event Hub, Service Bus, Logic App, ADLS gen 2) and Python
• 2 + Years of experience in more than one object-oriented programming languages like React.JS, Jode.JS, HTML, JavaScript
• Proficiency with HTML, CSS, JavaScript/jQuery, local storage and cross browser compatibility
• Preferred– any experience working with Guidewire© Application
• Any experience in Power BI Report Development will be a plus

Education

• Bachelor’s degree in computer science, or IT or equivalent

Required Skills : CI/CD,PowerBI,SSIS Additional Skills : Cloud Administrator,AWS Engineer,Data Engineer