Data Engineering Director

Job Description

• Translate business strategy into technical execution, ensuring data platforms enable both real-time and batch analytics at scale.

• Lead, mentor, and grow teams of data engineers and engineering managers with diverse experience levels.

• Acting as a lead/SME to the team building large scale, real time data pipelines with spark on Databricks.

• Working with cross functional teams and analysts to understand business requirements leverage our data.

• Translate complex business requirements into scalable, high-impact data solutions

• Build and align teams through resource planning, team member allocation, and
staffing projections

• Define and communicate a forward-looking technical roadmap for Data Engineering spanning multiple quarters to one year

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

• 5+ years of experience Spark, working in SparkSQL and/or DataFrames/Datasets API (with emphasis on DataFrames) to query and perform data manipulation

• 5+ years experience functioning as a lead or senior resource on a team.

• Experience building large scale Spark applications, ideally with either Batch processing and/or Streaming processing

• 3+ Python would be ideal but a solid knowledge of Scala is also acceptable

• 4+ Experience with cloud computing platforms, we use AWS (Kinesis, S3, Lambda)

• Has experience with ANSI SQL relational database (Oracle, SQL, Postgres, MySQL)

• Linux common working knowledge, including navigating through the file system and simple bash scripting

• General knowledge of distributed systems and distributed data processing frameworks

• Knowledge about agile software processes