Snowflake Data Engineer

Data Engineer

As a member of the Data Platforms Team, the Data Engineer role plays a critical part in defining the future Data Strategy for the BPX organization. This role is responsible for the development, guidance, and oversight for data ingestions and integrations for BPX data platforms. The Data Engineer works with the IT portfolio leads, data analytics teams, data scientists, data virtualization, data governance, data quality, DataOps Support, External Data Engineering consultants, and the Enterprise Data Architect to understand and provides direction on approved patterns for our Data Platforms to meet the project data requirements.

The Data Engineer must have strong SQL development knowledge to provide support to project developers/users with database design, data flow and analysis activities. The position of the Data Engineer also plays a key role in the development and deployment of innovative big data platforms for advanced analytics and data processing. The Data Engineer defines and builds the data rules and pipelines that will enable faster, better, data-informed decision-making within the business. The Data Engineer ensures stable solutions, stable infrastructure, and participates in the design and build of services, integrations, and ELT solutions.

This role has a few main objectives which include data platform architecture(50%), complex data integration project work(25%), definition of data standards/policies (5%), and responsibility to ensure Data Platform directions are followed by performing oversight for all Data Engineering projects(20%).

Key accountabilities

• Assembling large, complex sets of data that meet non-functional and functional business requirements
• Identifying, designing, and implementing internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes
• Building required infrastructure for optimal extraction, transformation and loading of data from various data sources
• Working with stakeholders including the Executive, Product, Data and Business teams to support their data infrastructure needs while assisting with data-related technical issues
• Engaging with Product Vendors to evaluate tools that support cloud based data platform needs, tools to speed up data delivery capabilities, and self service products for our resident developers
• Provide recommendation for new data platform products to Modernize our Cloud Data Platforms.
• Identify product or functionality gaps in our Data Platform capabilities and provide input on solutions to solve for those gaps
• Creating databases optimized for performance, implementing schema changes, and maintaining data architecture standards across BPX databases.
• Designing and developing scalable ETL/ELT packages from the business source systems.
• Enabling and running data migrations across different databases and different servers.
• Responsible for performing thorough testing and validation to support the accuracy of data transformations and data verification.
• Ensure proper data governance and quality across the Data Platform development areas.
• Performs ad-hoc analyses of data stored in the databases and designs, develops and tests SQL scripts, stored procedures, functions, and views
• Troubleshoots data issues within the business and across the business and presents solutions to these issues
• Analyze complex data elements and systems, data flow, dependencies, and relationships to contribute to & define conceptual physical and logical data models
• Be knowledgeable of industry trends and best practices, advising senior management on new and improved data engineering strategies.
• Drive excellent, consistent customer service.
• Ensure the proper management, maintenance and provisioning of all environments, databases and related technologies and solutions.

Essential experience and job requirements

• 5+ years of relevant work experience in IT/Data & Analytics
• 3 + years of Data Engineering experience
• Experience with AWS cloud preferred
• Experience with Snowflake preferred
• Experience in any big data technologies – Snowflake, Amazon Redshift, Azure Synapse
• Experience with ETL/ELT tools such as dbt, Stitch/Talend, AWS Glue, Azure Data Factory
• Experience with Replication tools such as FiveTran, Hevo, Qlik, HVR, etc
• Experience designing and delivering large scale, 24-7, mission-critical data pipelines and features using modern big data architectures
• Strong understanding of ETL/ELT processing with large data stores
• Strong data modelling skills (relational, dimensional, and flattened)
• Strong analytical and SQL skills, with attention to detail
• Validated experience with 1 or more non-SQL languages like Python or JavaScript
• Ability to work with multiple external teams and accomplish shared goals through the building consensus
• Strong communication (written/verbal) and collaboration skills
• Consulting, negotiation, and relationship skills
• Problem solving skills
• Enthusiastic, high-energy individual, self-motivated, people-oriented, and self-directed
• Must be able to maintain focus on achieving results, whilst being patient and pragmatic

Desirable criteria & qualifications

• Experience with the Azure and/or AWS cloud platforms
• Desire to continually learn outside of a classroom environment, and successfully apply learnings
• Demonstrated willingness to both teach others and learn new techniques

Required Skills : SNOWFLAKE; Very strong Python and SQL; AWS or Azure (bonus if they have both)Basic Qualification : Very senior data engineer with strong Python, SQL, knowledge of Snowflake and experience with cloud platforms.Additional Skills : Very senior data engineer with strong Python, SQL, knowledge of Snowflake and experience with cloud platforms.