Director of Data Engineering

Director Of Data Engineering

Measurement and data analysis lie at the foundation of our data products, but those measurements need to be assembled with context to turn them into actionable insights for our customers. As Director of Data Engineering, you will:

• provide technical leadership to the team responsible for building software and infrastructure for converting large scale raw measurement data into products that help customers make critical business decisions
• work closely with the product team, and the chief architect to scope out roadmap priorities
• oversee sprints for one or more geographically distributed engineering teams, and work with the product management team to ensure timely delivery of new features
• foster a collaborative and inclusive work environment

Your ideas and decisions will shape the next generations of:

• the Spark-based analysis algorithms that help us extract meaning from our raw measurements
• the metadata retrieval and transformation that allows us to correlate the events we observe with the reality of the real world
• the data warehouses where we store and query this information to derive new insights to guide our customers' decisions
• the metadata indexing and workflow services that coordinate data flow and task execution among all the stages of our pipeline
• the product services that generate and deliver data to our customers
• and the compute and storage infrastructure that makes all of this possible

We believe strongly in working together with domain experts to build our software, and our data analysis pipeline is built in the common language of data science, which is Python. The back end analyses live on a combination of MapReduce tools (primarily Spark) and SQL databases (Postgres and Redshift). Most of our backend services, and all of our APIs are implemented in Haskell.

The ideal candidate has led the development and delivery of data intensive products, and has experience building highly effective teams in the process. This candidate will understand how to manage the tradeoffs between delivering functionality for today, and building successful products for the long haul. The specific requirements are as follows:

• 8+ years of building and delivering software/data products (either as a leader or in hands on capacity)
• Experience leading and/or managing technical teams, and growing successful product teams
• Experience leading product teams that leverage a similar tech stack to us would be a big plus:
• Apache Spark
• Column-oriented SQL data warehouses such as Snowflake or Redshift
• Workflow orchestration tools such as Apache Airflow
• Machine learning frameworks
• ETL tools such as Pentaho or AWS Data Pipelines
• AWS devops tools and techniques
• Experience working with Machine Learning teams would be a big plus