Senior Data Engineer

Role Description

Allocate is looking for a Senior Data Engineer to help build out the data infrastructure that powers our analytics, reporting, and data-driven product features. In this role, you will partner closely with our data lead to model core financial entities, integrate internal and external sources, and build the pipelines and infrastructure that let our engineering and product teams make informed decisions and ship compelling features. This is a fully remote position where you will work alongside our backend team (C#/.NET) and frontend team (Node/Vue.js) to integrate data pipelines into our platform.

Responsibilities

• Build and Extend Data Architecture:
  • Build on and extend Allocate's data lakehouse on AWS.
  • Contribute to our knowledge graph that models key relationships (investors, funds, companies, etc.).
  • Integrate vector database for semantic search and retrieval across AI agents, models, and providers.
• Develop Data Pipelines:
  • Create robust ETL/ELT pipelines to ingest, clean, and transform data.
  • Ensure both batch processing and real-time data streaming are handled.
  • Build pipelines with scalability and reliability in mind.
• Enable AI/ML Capabilities:
  • Work closely with data science and engineering team to provision data and infrastructure for machine learning models.
  • Prepare training datasets and set up feature stores.
  • Orchestrate workflows that feed LLM-based agents with the necessary context.
• Engineering Excellence and Collaboration:
  • Partner with data lead and engineering team to deliver data and AI infrastructure.
  • Raise the bar through code review, testing, and adherence to best practices.
  • Work in cross-functional squads to incorporate data-driven features into the product roadmap.
• Infrastructure and DevOps:
  • Collaborate with DevOps engineers to deploy and maintain data services.
  • Containerize and orchestrate data tools using Docker/Kubernetes on AWS EKS.
  • Implement CI/CD pipelines for data workflows.
  • Monitor the health and performance of data platforms.
• Continuous Improvement:
  • Stay up to date with the latest in data engineering and AI.
  • Evaluate and recommend new technologies to improve pipeline reliability.
  • Encourage rethinking how things are done to build a world-class platform.

Qualifications

• 5+ years of hands-on experience in data engineering or related fields.
• Strong experience working with AWS cloud services for data.
• Proficiency in SQL and relational database design.
• Fluency in at least one major programming language used in data engineering.
• Understanding of machine learning models and data consumption.
• Solid understanding of containerization and deployment.
• Strong analytical and problem-solving skills.
• Experience working in a regulated SEC environment.
• Excellent communication skills and a collaborative mindset.
• Bachelor’s degree in Computer Science or equivalent practical experience.

Requirements

• Experience with tools like S3, EC2, ECS, EKS, Athena, Redshift, Glue, and Step Functions.
• Experience with data warehouses or lakehouses (e.g., Snowflake, Databricks Delta Lake).
• Experience with graph databases and knowledge graph schemas.
• Experience with Docker and Kubernetes for running distributed jobs/services.
• Experience with workflow managers (Airflow, Prefect, dbt, or similar).

Benefits

• Medical, dental, and vision insurance.
• 401(k) plan.
• Responsible vacation time (PTO).
• Travel required for team/department offsites.
• A Broadband internet connection required.