Data Engineer

Data Engineer

Remote

Clearance : Active Secret clearance is required.

Key Responsibilities

• Analyze science, engineering, business, and other data processing problems to develop and implement solutions to complex applications problems, system administration issues, or network concerns.
• Perform systems management and integration functions, improve existing computer systems, and review computer system capabilities, workflow, and schedule limitations.
• May analyze or recommend commercially available software.
• Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices.
• Troubleshoot data-related problems and provide innovative solutions to address complex data challenges.
• Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes.
• Leverage big data technologies to efficiently process and analyze large datasets.
• Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy and integrity of large datasets.
• Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing.
• Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures.
• Design, develop, and implement end-to-end data pipelines, utilizing ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL.

Required Qualifications

• 3–10 years of directly related experience and B.A./B.S. or M.A./M.S. in a related field.
• Experience building within Palantir Foundry applications, including Workshop, Ontology Manager, and Gaia.
• Experience with Machine Learning models and/or Generative AI.
• Experience with APIs.
• Previous experience with Advana or the War Data Platform.
• Experience with the DoW mission and associated datasets.
• Understanding of data modeling/visualization, database design principles, and data governance practices.
• Experience working with large datasets and leveraging big data technologies to process and analyze data efficiently.
• Proficiency in ETL processes and firsthand experience with technologies, such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks.
• 3+ years of experience as a Data Engineer.

Preferred Qualifications

• Experience with Machine Learning models and/or Generative AI.
• Experience with APIs.
• Previous experience with Advana or the War Data Platform.
• Experience with the DoW mission and associated datasets.
• Understanding of data modeling/visualization, database design principles, and data governance practices.
• Experience working with large datasets and leveraging big data technologies to process and analyze data efficiently.
• Proficiency in ETL processes and firsthand experience with technologies, such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL.