Sr. Data Engineer

Job Title: Data Engineer Location: Bethpage, NY Key Responsibilities: • Data Pipeline Design and Implementation: Develop and implement scalable and reliable data pipelines using GCP services like BigQuery, Dataflow, Cloud Storage, and Composer. • Medallion Architecture Implementation: Design and implement the bronze, silver, and gold layers within the Medallion architecture to organize and process data effectively. • Data Storage and Processing: Implement data storage solutions and optimize data processing workflows using appropriate GCP tools and technologies. • Data Quality and Security: Ensure data quality, integrity, and security throughout the data lifecycle, including data validation, cleansing, and access control. • Collaboration and Communication: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective data solutions. • Monitoring and Maintenance: Monitor and maintain the health of the data infrastructure, troubleshoot and resolve data-related issues, and ensure optimal performance. • Staying Updated: Keep up to date with the latest GCP features, data engineering best practices, and advancements in data architecture. • Data Governance: Implement data governance strategies to ensure compliance with industry standards and best practices. Skills and Qualifications: • GCP Expertise: Strong knowledge of Google Cloud Platform (GCP) services, including BigQuery, Dataflow, Cloud Storage, and Composer. • Medallion Architecture: Solid understanding of the Medallion architecture principles and best practices. • Data Engineering: Proven experience in data engineering, including data warehousing, ETL processes, and data modelling. • Programming Languages: Proficiency in programming languages like Python, SQL, and potentially Apache Beam (for Dataflow). • Problem-Solving: Strong problem-solving and analytical skills to troubleshoot and resolve data-related issues. • Communication: Excellent communication and collaboration skills to work effectively with various teams. • Cloud Architecture: Knowledge of cloud architecture best practices, including scalability, reliability, and security. • Data Modelling: Experience with data modelling techniques and tools. In essence, a GCP Data Engineer focused on Medallion Architecture is a key player in building and managing data platforms on GCP, leveraging the Medallion pattern to ensure data quality, efficiency, and accessibility for various data consumers. Redolent