Principal Data Engineer

Job Description

This role is less of a traditional Data Engineer and more of a Platform Engineer / Data Engineering Lead focused on Foundry, governance, release engineering, and geospatial data products. The emphasis should shift away from building new models and toward owning, improving, governing, and operationalizing an existing platform and pipeline ecosystem.

We are seeking a Principal Data Engineer to support the transition of data pipelines from an internal IT team to our internal team.

This role will focus on both enhancing existing pipelines and improving the underlying code and data infrastructure to ensure long-term maintainability, scalability, and clarity.

Key Responsibilities

• Refactor and enhance data pipelines to be modular, maintainable, and well-documented

• Establish and promote best practices for code architecture, version control, and code management

• Collaborate closely with a core team of data scientists, machine learning engineers, and data engineers and a broader team of cross-functional partners

• Communicate technical data to non-technical stakeholders to build trust and understanding in the methodology.

• Support the development of machine learning models currently being developed in-house

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

Required Qualifications

• Strong experience in Python-based code development

• Demonstrated expertise in code architecture, software engineering practices, and maintainable pipeline design, in a cloud-based environment

• Experience with the Palantir Foundry platform (current technology, could move to AWS in future, as an option)

• Familiarity with PySpark and distributed computing.

• Experience with release engineering

• Repository governance experience; audit ready and branching strategy

• Traceability

• Experience working with spatial-temporal datasets and geospatial packages such as Sedona, Geopandas, and Rasterio.

• Experience transitioning data pipelines from consulting or external vendors into production environments

• Experience designing cloud-optimized geospatial datasets (GeoParquet, Parquet, Zarr) with efficient partitioning and support for large-scale spatial operations and aggregations.

• Proficiency with version control (Git) and collaborative workflows (e.g., pull requests, code reviews)

• Ability to assess and improve legacy or externally developed codebases

• Excellent communication skills

Preferred Qualifications

• Experience with a GIS platform such as QGIS or ArcGIS

• Experience working with ML model outputs

• Experience designing production-grade data and ML platforms with strong emphasis on reproducibility, dataset versioning, release traceability, repository governance, and maintainable software architecture.