Senior Data Engineer

Position Overview


Index Analytics is seeking a Sr. Data Engineer to support Government clients to design, build, and optimize scalable cloud-based solutions, data pipelines, and implement connections to knowledge sources. The Sr. Data Engineer plays a key role in modernizing the organization's data ecosystem by helping transition legacy solutions to a contemporary infrastructure.


As part of a cross-functional team including Data Engineers, health policy researchers, Analysts, the engineer will support efforts to design and implement a robust environment capable of ingesting diverse data sources to support advanced analytics and reporting needs. Core responsibilities include defining structural, interface, and business requirements for data solutions; designing relational and non-relational databases and their associated integration components; and implementing Python based automated data pipelines.

This role blends advanced data engineering with hands-on cloud solutions engineering, leveraging AWS, Databricks and modern DevOps practices. The ideal candidate has experience delivering high-quality solutions in an Agile environment.

Responsibilities

• Collaborate closely with stakeholders, cross-functional and internal technical teams to gather requirements, document business rules and develop a thorough understanding of the business context and objectives.
• Configure and manage connections from analytic tools to back end data sources, repositories, and platforms including Databricks and various APIs.
• Oversee Databricks unity catalog population and administration.
• Formulate and document technical and coding standards.
• Collaborate to design secure, scalable, and cost-optimized data solutions.
• Design, build, and maintain scalable, reliable ETL/ELT data pipelines using AWS and Databricks.
• Develop and optimize data models, both conceptual and physical, to support analytics, reporting, and operational consumption.
• Implement data quality, validation, and monitoring frameworks to ensure accuracy and reliability.
• Ensure data workflows are modular, testable, and properly version-controlled.
• Operationalize pipelines with monitoring, alerting, and automated recovery mechanisms.
• Conduct advanced data analysis using languages such as Python and SQL.
• Develop documentation to include data models, data dictionaries, and data usage guides.
• Improve end-to-end performance of data workflows.
• Build and maintain CI/CD pipelines using GitHub to support automated testing, deployments, and continuous integration.
• Meet schedule deadlines and commitments with a high-level of quality of deliverables.
• Collaborate with a team of cross-functional resources in an Agile delivery environment to deliver iterative value.

• U.S. Citizen or authorized to work in the United States and have resided in the U.S. for at least three of the past five years. Must be eligible to obtain a federal government client badge and successfully pass a government background investigation.
• Bachelor's degree and a minimum of 10 years of professional experience, including at least 4 years designing, developing, and implementing Business Intelligence (BI), data management, analytics, or similar enterprise IT solutions. An equivalent combination of education and experience may be considered. Four years of specialized experience may substitute for a bachelor's degree.
• Demonstrated expertise in data engineering, data integration, and enterprise data platform development, with hands-on experience designing and implementing scalable data pipelines, data warehouses, data lakes, and cloud-based data solutions.
• Advanced proficiency in SQL and Python, including experience developing, optimizing, testing, and maintaining data transformation processes, data pipelines, and analytical workflows.
• Strong experience with AWS cloud services and Databricks, including the design, implementation, and support of modern data architectures. Experience with artificial intelligence, machine learning, or generative AI technologies is highly desirable.
• Proven experience using GitHub for source control management, branching strategies, code reviews, versioning, and collaborative software development practices.
• Strong understanding of modern data architecture principles, including ETL/ELT frameworks, data modeling, data governance, data integration patterns, relational and non-relational databases, and enterprise data warehousing concepts.
• Experience applying software engineering best practices, including object-oriented programming, design patterns, code quality, testing, documentation, and maintainable solution development.
• Familiarity with DevOps methodologies and CI/CD pipelines, including experience with automated build, deployment, testing, and monitoring tools.
• Strong analytical, problem-solving, and communication skills, with the ability to translate complex technical concepts into actionable insights and effectively present findings and recommendations to technical and non-technical stakeholders.
• Excellent written and verbal communication skills, including the ability to collaborate across multidisciplinary teams, facilitate discussions, develop technical documentation, and provide executive-level reporting.
• Working knowledge of Agile frameworks and Scrum methodologies, including experience using tools that support Agile delivery, sprint planning, backlog management, and team collaboration.
• Experience working with CMS enterprise repositories, including the Integrated Data Repository (IDR), is preferred.
• Prior experience supporting federal government agencies, healthcare programs, or large-scale public sector data environments is preferred.