Data Integration Engineer

Key Qualifications

MUST HAVE:
• Snowflake Data Engineering


o Design and implement enterprise-grade data pipelines using Snowflake, including ingestion and transformation


o Must be strong in both Core and Semantic aspects


o Develop complex SQL transformations, stored procedures and Dynamic tables inside Snowflake to enable near real-time and batch processing


o Implement Snowflake data sharing, data marketplace integrations


o Engineer Snowpipe and Kafka-to-Snowflake streaming ingestion pipelines also handling high throughput event data at scale


o Optimize Snowflake cluster performance virtual warehouse sizing, query profiling, clustering keys


o Architecture, design aspects, performance tuning, time travel, warehouse concepts - scaling, clustering, micro-partitioning


o Experience with SnowSQL, Snowpipe
• Data Integration aspects


o Design and maintain end-to-end ETL/ELT pipelines using Apache Airflow


o Experience in building reusable parameterized data ingestion pipelines/frameworks is beneficial.


o Thorough on data quality checks
• AI and Data Science


o Integrate AI/LLMs with data pipelines via Python UDFs or API callouts enabling text analytics, semantic search and GEN-AI augmented workflows


o Experience with Python based frameworks scikit learn, PyTorch, TensorFlow


o Experience with NLP and text-mining techniques on unstructured data to identify actionable information


o Time-series forecasting, anomaly detection and propensity modeling
• Experience with Data Visualization aspects
• Hands-on experience with writing Complex queries using Joins, Self Joins, Views, Materialized Views, Cursor also Recursive, use of GROUP BY, PARTITION BY functions / SQL Performance tuning
• Hands-on experience with ETL and Dimensional Data Modelling Slowly Changing Dimensions (SCD Type 1, 2, 3)


o Good understanding of concepts like schema types, table types - fact-dimension etc. like how to design a dimension vs fact, design considerations factored etc.
• Proficiency in Python scripting/programming using Pandas, PyParsing, Airflow.


o Pandas, Tableau server modules, Numpy, Datetime, Apache Airflow related modules, APIs


o Data Pipeline automation


o Strong Python programming skills
• Actively participating in discussions with business to understand requirements, perform thorough impact analysis and provide suitable solutions.

Key Words to search in Resume


Snowflake, Advanced SQL, Dimensional Data Modelling (Slowly Changing Dimensions), Python, AI, Data Science, Data Visualization

Location: Sunnyvale, CA


Salary range :$80,000-$140,000 a Year.


#LI-AS3