Data Engineer

Job Description

Data Engineer

\n

Remote

\n

\n

Clearance : Active Secret clearance is required.

\n

\n

Key Responsibilities

\n

\n
• Analyze science, engineering, business, and other data processing problems to develop and implement solutions to complex applications problems, system administration issues, or network concerns.
\n
• Perform systems management and integration functions, improve existing computer systems, and review computer system capabilities, workflow, and schedule limitations.
\n
• May analyze or recommend commercially available software.
\n
• Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices.
\n
• Troubleshoot data-related problems and provide innovative solutions to address complex data challenges.
\n
• Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes.
\n
• Leverage big data technologies to efficiently process and analyze large datasets.
\n
• Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy and integrity of large datasets.
\n
• Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing.
\n
• Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures.
\n
• Design, develop, and implement end-to-end data pipelines, utilizing ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL.
\n

\n

\n

Required Qualifications

\n

\n
• 3–10 years of directly related experience and B.A./B.S. or M.A./M.S. in a related field.
\n
• Experience building within Palantir Foundry applications, including Workshop, Ontology Manager, and Gaia.
\n
• Experience with Machine Learning models and/or Generative AI.
\n
• Experience with APIs.
\n
• Previous experience with Advana or the War Data Platform.
\n
• Experience with the DoW mission and associated datasets.
\n
• Understanding of data modeling/visualization, database design principles, and data governance practices.
\n
• Experience working with large datasets and leveraging big data technologies to process and analyze data efficiently.
\n
• Proficiency in ETL processes and firsthand experience with technologies, such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks.
\n
• 3+ years of experience as a Data Engineer.
\n

\n

\n

Preferred Qualifications

\n

\n
• Experience with Machine Learning models and/or Generative AI.
\n
• Experience with APIs.
\n
• Previous experience with Advana or the War Data Platform.
\n
• Experience with the DoW mission and associated datasets.
\n
• Understanding of data modeling/visualization, database design principles, and data governance practices.
\n
• Experience working with large datasets and leveraging big data technologies to process and analyze data efficiently.
\n
• Proficiency in ETL processes and firsthand experience with technologies, such as Databricks, Python, Spark, Scala, JavaScript/JSON, and SQL.
\n

\n