Sr. Data Engineer

Job Title: Data Engineer


Location: Bethpage, NY


Key Responsibilities:
• Data Pipeline Design and Implementation:


Develop and implement scalable and reliable data pipelines using GCP services like BigQuery, Dataflow, Cloud Storage, and Composer.
• Medallion Architecture Implementation:


Design and implement the bronze, silver, and gold layers within the Medallion architecture to organize and process data effectively.
• Data Storage and Processing:


Implement data storage solutions and optimize data processing workflows using appropriate GCP tools and technologies.

• Data Quality and Security:


Ensure data quality, integrity, and security throughout the data lifecycle, including data validation, cleansing, and access control.
• Collaboration and Communication:


Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective data solutions.

• Monitoring and Maintenance:


Monitor and maintain the health of the data infrastructure, troubleshoot and resolve data-related issues, and ensure optimal performance.
• Staying Updated:


Keep up to date with the latest GCP features, data engineering best practices, and advancements in data architecture.

• Data Governance:


Implement data governance strategies to ensure compliance with industry standards and best practices.

Skills and Qualifications:
• GCP Expertise:


Strong knowledge of Google Cloud Platform (GCP) services, including BigQuery, Dataflow, Cloud Storage, and Composer.
• Medallion Architecture:


Solid understanding of the Medallion architecture principles and best practices.
• Data Engineering:


Proven experience in data engineering, including data warehousing, ETL processes, and data modelling.
• Programming Languages:


Proficiency in programming languages like Python, SQL, and potentially Apache Beam (for Dataflow).

• Problem-Solving:


Strong problem-solving and analytical skills to troubleshoot and resolve data-related issues.

• Communication:


Excellent communication and collaboration skills to work effectively with various teams.
• Cloud Architecture:


Knowledge of cloud architecture best practices, including scalability, reliability, and security.
• Data Modelling:


Experience with data modelling techniques and tools.

In essence, a GCP Data Engineer focused on Medallion Architecture is a key player in building and managing data platforms on GCP, leveraging the Medallion pattern to ensure data quality, efficiency, and accessibility for various data consumers.