Sr. Data Engineer

Data Engineer

Location: Bethpage, NY

Key Responsibilities:

• Data Pipeline Design and Implementation: Develop and implement scalable and reliable data pipelines using GCP services like BigQuery, Dataflow, Cloud Storage, and Composer.
• Medallion Architecture Implementation: Design and implement the bronze, silver, and gold layers within the Medallion architecture to organize and process data effectively.
• Data Storage and Processing: Implement data storage solutions and optimize data processing workflows using appropriate GCP tools and technologies.
• Data Quality and Security: Ensure data quality, integrity, and security throughout the data lifecycle, including data validation, cleansing, and access control.
• Collaboration and Communication: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective data solutions.
• Monitoring and Maintenance: Monitor and maintain the health of the data infrastructure, troubleshoot and resolve data-related issues, and ensure optimal performance.
• Staying Updated: Keep up to date with the latest GCP features, data engineering best practices, and advancements in data architecture.
• Data Governance: Implement data governance strategies to ensure compliance with industry standards and best practices.

Skills and Qualifications:

• GCP Expertise: Strong knowledge of Google Cloud Platform (GCP) services, including BigQuery, Dataflow, Cloud Storage, and Composer.
• Medallion Architecture: Solid understanding of the Medallion architecture principles and best practices.
• Data Engineering: Proven experience in data engineering, including data warehousing, ETL processes, and data modelling.
• Programming Languages: Proficiency in programming languages like Python, SQL, and potentially Apache Beam (for Dataflow).
• Problem-Solving: Strong problem-solving and analytical skills to troubleshoot and resolve data-related issues.
• Communication: Excellent communication and collaboration skills to work effectively with various teams.
• Cloud Architecture: Knowledge of cloud architecture best practices, including scalability, reliability, and security.
• Data Modelling: Experience with data modelling techniques and tools.

In essence, a GCP Data Engineer focused on Medallion Architecture is a key player in building and managing data platforms on GCP, leveraging the Medallion pattern to ensure data quality, efficiency, and accessibility for various data consumers.