Lead Data Engineer

Title: Lead Data Engineer
Location: Hybrid in SF (Tuesdays onsite)
Openings: 1
Work Schedule: (available until 12 am PT time to overlap with onshore team)
Follow-Up Meeting: After each interview is scheduled.
Contract Type: 12 months contract extensions

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to View email address on click.appcast.io learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:


Required Skills & Experience

Must Haves:
• At least 5 years of strong handson expertise with Databricks and Datastage (or comparable ETL tools)
• Experience with migrations
• Exs: Migrating SQL to cloud, SQL to Azure Databricks, (on prem to the cloud)
• 8-10 years of software development and data engineering experience with SQL, ADF, SSAS cubes, Cognos, Tableau, ThoughtSpot and other BI tools
• Advanced SQL skills for processing raw data, Kafka ingestions, ADF pipelines, data validation and QA with experience across SQL and NoSQL databases
• Programming experience with Python and/or Scala
• Experience ingesting data via APIs and streaming sources
• Experience building data pipelines, data warehouses, and analytics platforms
• Proficiency with Spark, Hive, Kafka, and cloud data services
• Experience with Azure, AWS or equivalent cloud platforms

Day-To-Day:
• Lead and support an offshore engineering team as the primary onshore technical owner
• Define and drive engineering strategy in partnership with product and technology leaders
• Own the migration from SQL/Datastage to Databricks, with full understanding of the data landscape
• Review existing Datastage pipelines and reengineer them in Databricks
• Provide handson mentorship and technical guidance
• Develop and deploy solutions using cloudbased platforms daily


Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.