Data Engineer

Data EngineerThe American College of Physicians (ACP) is looking for a Data Engineer to join the Technology division. ACP, a national organization of internal medicine physicians with headquarters in Philadelphia, is the largest medical specialty organization and the second-largest physician group in the United States. This is a full-time hybrid position with 2 days in-office (Philadelphia HQ) per week required.The Data Engineer will provide technical leadership responsibility for building and managing enterprise-wide data transformation and orchestration processes and data pipelines. This work will require expertise in several layers of our enterprise data infrastructure, including our diverse source systems, our medallion-modeled data lakehouse, data activation platforms, and the orchestration and transformation of data as it flows throughout the entire data estate. Responsibilities include but are not limited to:Provides technical leadership in data pipeline design and development organization-wide, ensuring data flows are orchestrated in alignment with business needs and system architectures. Emphasis will be on ingestion, unifying transformations, and deliveries to activation platforms via the data lakehouse.Ensures that designs and techniques are compliant with system architectures as well as data management and governance standards and policies in place within the organization.Uses agile software development processes, in collaboration and alignment with the development, analyst, and data administration teams, to make iterative deployments with full visibility and change tracking.Ensures documentation of data models, transformations, and orchestration remain current, comprehensive, semantically precise, and accessible to all staff.Partners with others across the data management team to collaborate with business stakeholders, ensuring that pipelines are optimized for quality, reliability, contextual accuracy, and actionability.Plays a key role in enterprise data architecture design and analysis, including interactions with consultants and contractors.Monitors data systems and flows, utilizing automated tooling to ensure successful and reliable pipelines.Qualifications:A Bachelor's degree is required, preferably in Computer Science, Software Engineering, Systems Engineering, Data Science, or another field closely related to data engineering.Demonstrable experience with the Snowflake platform and related tooling, e.g., dbt and Fivetran, is required. Currency in the evolution of Cortex Code (CoCo) and Semantic Views is highly desirable.At least 3 years of experience designing and developing data pipelines in modern data engineering environments is required. Experience working with large-scale (1M+ records) customer and behavioral data models and systems, e.g., "Customer 360" models highly desirable.Experience with multiple techniques for modeling and transformations, e.g., ETL, reverse ETL, and modern ELT techniques/platforms, as well as real-time/streaming processes, e.g., Apache Kafka.Strong understanding of all stages of integrated, data lakehouse-oriented architectures, from ingestion to semantic modeling to delivery/activation.Deep experience with core data engineering coding techniques/languages: SQL, Python.Experience working within multiple large-scale API-based, cloud architectures, e.g., Amazon AWS-based and Microsoft Azure-based is highly preferred.Excellent oral and written communication skills are essential to interact with all levels of College staff, members, and external resources in both group and one-on-one settings.ACP offers a competitive salary, superior benefits and a supportive work environment.