Data Engineer

Data Engineer

The National Resident Matching Program® (NRMP®) is a private, not-for-profit organization established in 1952 to provide a uniform date of appointment to positions in graduate medical education (GME) in the United States. The NRMP is not an application processing service; rather, it provides an impartial venue for matching applicants' and programs' preferences for each other consistently. The NRMP conducts the annual Main Residency Match encompassing 40,000 applicants for more than 25,000 positions in core residencies as well as Matches for fellowship positions in 55 subspecialties through its Specialties Matching Service®.

NRMP is looking to fill the full-time position of data engineer. The data engineer will play a pivotal role in operationalizing the data and analytics initiatives for NRMP. The data engineer will be building, managing, integrating, and optimizing reusable data pipelines and moving these data pipelines effectively into production for key data and analytics consumers - business/data analysts, research analysts, executives, and internal departments - that need curated data sets for data and analytics use cases.

The data engineer must support compliance with data governance and data security and privacy requirements to enable faster data access, integrated data reuse and vastly improved time-to-solution for NRMP data and analytics initiatives. The data engineer will be measured on their ability and agility to integrate analytics results with NRMP's business processes.

This role will be the key interface in operationalizing data and analytics on behalf of organizational outcomes and will require both creative and collaborative aspects- working with both IT and the wider business. The data engineer will also be tasked with working with key business stakeholders, IT experts and subject-matter experts to plan and deliver optimal analytics solutions.

Essential Duties And Responsibilities

• Build data pipelines: Managed data pipelines consist of a series of stages through which data flows from data sources of acquisition to integration to consumption for specific use cases. These data pipelines must be created, maintained, and optimized as workloads move from development to production for specific use cases. Architecting, creating, and maintaining data pipelines will be the primary responsibility of the data engineer.
• Drive Automation through effective metadata management: The data engineer will be responsible for using innovative and modern tools, techniques, and architectures to partially or completely automate the most-common, repeatable and tedious data preparation and integration tasks in order to minimize manual and error-prone processes and improve productivity. The data engineer will also need to assist with renovating the data management infrastructure to drive automation in data integration and management.

This will include (but not be limited to):

• Using modern data preparation, integration and metadata management tools and techniques.
• Tracking data consumption patterns.
• Monitoring schema changes.
• Recommending and automating — existing and future integration flows.
• Collaborate across departments: The newly hired data engineer will need strong collaboration skills to work with varied stakeholders within the organization. In particular, the data engineer will work in close relationship with research teams and with data analysts in refining their data requirements for various data and analytics initiatives and their data consumption requirements.
• Become a data and analytics evangelist: The data engineer will be considered a blend of data and analytics "evangelist," "data guru" and "fixer." This role will promote the available data and analytics capabilities and expertise to business unit leaders and educate them in leveraging these capabilities in achieving their business goals.

Education, Experience And General Requirements

• At least 6-8 years or more of work experience in data management disciplines including data integration, modeling, optimization, and data quality, and/or other areas directly relevant to data engineering responsibilities and tasks.
• At least 3 years of experience working in cross-functional teams and collaborating with business stakeholders in support of departmental and/or multi-departmental data management, analytics, and business intelligence initiatives.
• A bachelor's or master's degree in engineering or computer science or a related quantitative field.
• AWS Certified Developer certification is highly desirable.
• The ideal candidate will have a combination of IT skills, data governance skills, and analytics skills.
• Familiarity with undergraduate and graduate medical education and the residency selection process is highly desirable.
• Legal authorization to work in the United States without sponsorship or restriction.
• Resides in the United States and ability to work remotely with occasional overnight travel.

Technical Knowledge And Skills

• Strong experience with Object-oriented/object function scripting using python and related libraries.
• Strong experience with popular database programming languages including SQL and PL/SQL for relational databases.
• Strong experience in working with and optimizing ETL/ELT processes and data integration / data preparation flows and moving them across various environments including production.
• Proficient in working in AWS environment (Glue, S3, Lambda, IAM)
• Experience in working with open-source technologies such as Airflow to automate data pipelines.
• Adept in agile methodologies and capable of applying DevOps and increasingly DataOps principles to data pipelines to improve the communication, integration, reuse and automation of data flows between data providers and consumers across NRMP.
• Ability to implement data quality checks and ensure data integrity within the data warehouse environment
• Experience working with data quality, security, and governance teams in moving data pipelines through environments with appropriate data quality, governance and security standards.

Interpersonal Skills And Characteristics

• Be highly creative and collaborative. An ideal candidate would be expected to collaborate with cross functional teams to define the business problem, refine the requirements, and design and develop data deliverables accordingly.
• Be a confident, energetic self-starter with strong interpersonal skills.
• Comfortable in a fast-paced small company environment with the ability to manage a variety of projects simultaneously.
• Have good judgment, demonstrate initiative, and demonstrate commitment to high standards of ethics, regulatory compliance, customer service, and business integrity.
• Collaborate with Business Intelligence team to build effective solutions
• Keen interest in learning and using latest software tools, methods and technologies to solve problems with an eye on maintainability.
• Be a strategic, intellectually curious thinker with a focus on outcomes.

Interactions

External:

• Frequent interaction with NRMP IT consultants and partners.
• Periodic interaction with staff of other national health care organizations and researchers in academia.

Internal:

• Frequent interaction with other NRMP staff in monitoring data warehouse, data pipelines, research and reporting environments

ADA Specifications

This position is primarily remote and sedentary; there may be infrequent overnight travel and duties may require some bending and lifting. The above statements describe the general nature and level of work being performed by the individual(s) assigned to this position. They are not an exhaustive list of all duties, responsibilities, and skills required.

NRMP is an equal opportunity employer and values diversity. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, disability or other protected class status. In addition, reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions of this position.