Lead Data Engineer (Azure Databricks, PySpark & Spark Optimization) - Canada (Remote)
Saransh Inc
Role: Lead Data Engineer (Azure Databricks, PySpark & Spark Optimization)
Remote (Canada)
Job Type: Contract
Role Summary:
- We are looking for a hands-on Lead Data Engineer to support a proof of concept in Azure Databricks.
- The role focuses on converting complex SQL stored procedures into optimized Python and PySpark code. The objective is to demonstrate that the platform can process 100,000 simulation years in minutes rather than hours, while maintaining accurate results.
Key Responsibilities:
-
Refactor complex T-SQL logic , including pricing rules, layer applications, and reinstatements, into Python and PySpark .
-
Replace row-by-row processing with efficient, vectorized DataFrame operations.
-
Develop Azure Data Factory pipelines to ingest loss data from Touchstone/RMS into ADLS Gen2.
-
Optimize Spark jobs through effective partitioning, memory management, and skew handling.
-
Validate the new solution against existing SQL results to ensure accuracy.
-
Contribute to performance testing and troubleshooting.
Required Skills:
-
Advanced Python skills , including Pandas and PySpark .
-
Strong T-SQL experience and the ability to analyze complex stored procedures.
-
Hands-on experience with Azure Databricks, Delta Lake, and Azure Data Factory .
-
Strong knowledge of distributed computing and Spark optimization .
-
Experience with partitioning, sharding, broadcast variables, and data-skew management.
-
Familiarity with CI/CD and version control for data engineering projects.
Desirable skills:
-
Experience with Monte Carlo simulations or stochastic modelling .
-
Knowledge of Parquet and Avro file formats.
-
Experience analyzing SQL Server execution plans.
-
Background in insurance, risk analytics, or simulation-based modelling .
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Lead Data Engineer (Azure Databricks, PySpark & Spark Optimization) - Canada (Remote) in Scotia, NY vacancy
$59.5 per hour
...Must Haves: -Azure -Databricks -Apache Spark -CI/CD Databricks... ...the Databricks data platform. The... ...focus on data engineering, integration,... ...with Python, PySpark, SQL, and/or Java... ...and optimization of data processing... ...the U.S., UK and Canada. Join Yoh’s extensive...DatabricksYoh, A Day & Zimmermann Company
Jersey City, NJ3 days ago- ...Programming & Data Engineering Expert - level proficiency... ...Python, Scala, and PySpark, with a strong... ...troubleshooting Spark workloads; and optimizing large-scale batch... ...contract definition. Databricks expertise Delta... ...Serving. Azure data stack — ADLS Gen...Remote workDatabricksContract work
Reuben Cooley Inc.
Cary, NC1 day ago - ...seeking a skilled Senior Data Software Engineer with strong expertise in PySpark , SQL , and unit... ...with Apache Spark , preferably within the Databricks environment, and will... ...scalable data pipelines, optimizing data workflows, and... ...platforms (e.g., Azure, AWS, Google Cloud...Databricks
- ...Job Title : Lead AWS Data Bricks Spark/Scala Engineer Job ID: 2022-11882 Job Location: Remote, Memphis, TN (preferred), Morris Plains, NJ, Austin, TX (preferred),... ...Eligibility to Work Required Key Technology : DataBricks, AWS, Spark Job Responsibilities :...Remote workDatabricksPermanent employment
Highbrow
Morris Plains, NJ1 day ago €38 per hour
...Senior Data Engineer (Databricks & PySpark) Location: 100% Remote (Warsaw, PL) Rate: Up to 38 EUR / hour... ...external systems. Performance Optimization: Optimize workloads for... ...Databricks (including Azure Databricks and... ...proficiency in PySpark, Python, Spark SQL, Data Modelling,...Remote workDatabricksFull timeOptiveum
New York, NY3 days ago- ...across US, Canada, and delivery... ...permanent remote working... ...end-to-end data architecture... ...ecosystem using Azure-native services and Databricks Build... ...Develop PySpark notebooks... ...architecture / data engineering roles ~... ...and Spark SQL Strong... ...Ability to optimize: Data...Remote workDatabricksPermanent employmentPart time
Exavalu
Dallas, TX3 days ago - ...Chicago, is seeking a Lead Data Engineer for a contract... ...implementation of a modern AWS + Databricks Lakehouse platform,... ..., with mostly remote work and the ability... ...governance, and cost optimization. Qualifications... ...experience with Databricks (Spark, Delta Lake, Unity Catalog...Remote workDatabricksContract workWork at office
Intone Inc
Chicago, IL4 days ago - ...ability to derive data-driven... ...dynamic and fully remote AI and Data... ...in the data engineering space to solve... ...to deliver optimal data engineering... ...Cortex, Databricks Genie, BI platforms... ...Python, and Spark, following... ...practices. As a Lead Data Engineer... ...(AWS, Azure, GCP) and container...Remote workDatabricks
Aimpoint Digital
Atlanta, GA16 hours ago $120.9k - $187k
...POSITION TITLELead Data EngineerThis role... ...role is open to remote candidates... ...governance, and data engineering from source systems... ...consumers—is seeking a Lead Data Engineer with strong Databricks expertise and... ...needed.Driving Spark performance optimization and leading platform...Remote workDatabricksFull timeWork at officeLocal areaWork visaGenworth Financial
Richmond, VA3 days ago$92k - $138k
Lead Data Engineer Lead II - Data EngineeringWho We Are:Born... ...architect, build, and optimize scalable, cloud-native... ...data platforms using Databricks and AWS.The... ...· Mastery of Python, PySpark, and advanced SQL for... ...below.Role Location: Remote - USCompensation Range...Remote workDatabricksFull timeTemporary workPart timeWork at officeLocal areaFlexible hoursUST Global
Chicago, IL2 days ago$170k - $240k
...We are seeking a Lead - Data Engineer to lead the hands-on... ...automation within an Azure Databricks environment. As... ...Databricks, Apache Spark, PySpark, SQL, and Delta Lake... .... Build and optimize data access layers... ...This position is remotely based in the U.S....Remote workDatabricksAstrana Health, Inc.
Monterey Park, CA7 days ago$161k - $266k
...Title and Summary Lead Data Engineer Overview:... ...focus heavily on Databricks, Snowflake, and AWS... ...architecture across AWS and Azure, enabling secure... ...excellence. Optimize data platforms... ...proficiency with Apache Spark/PySpark, Delta Lake, SQL,... ...In the US or Canada, if you require...DatabricksFull timePart timeImmediate startWorldwideFlexible hours3 days per weekMastercard Inc.
Arlington, VA1 hour ago$55 - $60 per hour
...26 Location: Remote; quarterly travel... ...Overview The Data Automation Engineer designs,... ..., and Apache Spark/PySpark. This role focuses... ..., pipeline optimization, and... ...AWS, Microsoft Azure, SQL databases... ...AWS, Azure, or Databricks certifications... ...Across the U.S., leading companies in...Remote workDatabricksHourly payFull timeContract workFor contractorsMonday to FridayShift workEquiliem
Washington DC3 days ago- ...Lead Data Engineer / Senior Data EngineerWe are seeking a highly skilled... ...experience in Data Engineering, Azure Databricks, and AI-enabled analytics... ...practices.Implement and optimize data ingestion, transformation... ...frameworks using PySpark, SQL, Python, and Azure Data...Remote workDatabricks
Georgia IT Inc
Chicago, IL5 days ago - ...Orchestration Engine helping... ...hiring aLead Data Engineerto drive... ...on technical lead responsible for... ...across multiple Databricks instances as... ...complex Spark and Databricks... ...Databricks, PySpark, and Spark SQL... ...skills in a remote environment... ...with sources in Azure, AWS, GCP, and...Remote workDatabricksFor contractorsWork at office
Logicbroker
New York, NY4 days ago - ...Lead, Data EngineerL3Harris is dedicated to... ...Melbourne, FL or Remote OpportunityJob Schedule... ...seeking a Data Engineer with experience... ...Palantir Foundry, Azure Data Factory,... ...Python, Pandas, Databricks, JavaScript, Typescript... ...leveraging PySpark and Spark SQLFamiliarity with...Remote workDatabricksLocal area
L3Harris Technologies
Melbourne, FL1 day ago $77.6k - $176k
...Job Number: R0251622 Databricks Data Engineer, Lead The Opportunity: As a... ...will include creating and optimizing end-to-end data flows, handling... ...writing Python and PySpark code for ETL scripts and custom... ...on during meetings. Remote : If this position is listed...Remote workDatabricksFull timeContract workPart timeLocal area- ...Truelogic is seeking a Data Engineer with 5–7 years of hands-on experience to join... ...large‑scale data processing using PySpark and Databricks. You will optimize performance, ensure data quality and... ...data solutions. This is a 100% remote role with USD pay. #J-18808-Ljbffr...Remote workDatabricks
TrueLogic Company
New York, NY1 day ago - ...OpenKrill is seeking a Data Engineer with 5–7 years of hands-on experience to... ...team. You will design, build, and optimize ETL pipelines using PySpark and Databricks, delivering scalable data... ...concepts clearly. This role is fully remote with USD pay and a strong emphasis...Remote workDatabricks
OpenKrill
United States1 day ago - ...Job Title: Data Engineer (Databricks, Azure, Kafka) Location: Remote Job Type: W2 / C2C / 1099 Job... ...using Databricks, Spark, and Azure cloud services... ...Design, develop, and optimize scalable data pipelines... ...Databricks and Spark (PySpark) Build and maintain ETL...Remote workDatabricks
- ...Truelogic is seeking a skilled Data Engineer with 5–7 years of hands-on experience... ...design scalable ETL pipelines, optimize performance, and collaborate... ...and business units in a fully remote capacity. The role emphasizes PySpark and Databricks proficiency, strong problem-...Remote workDatabricks
TrueLogic Company
New York, NY1 day ago $140k - $231k
...And Summary Lead Data Engineer Overview Mastercard... ...using Databricks, Apache Spark, Delta Lake, and... ...and services. Optimize large-scale data... ...Databricks, Apache Spark/PySpark, Delta Lake, SQL... ...platforms using Azure, AWS, or GCP.... .... In the US or Canada, if you require...DatabricksFull timePart timeWorldwideFlexible hours3 days per weekMasterCard
O Fallon, MO3 days ago- ...TrueLogic is seeking a skilled Data Engineer to join our client’s team. You will design, build, and optimize ETL pipelines, with hands-on work in PySpark and Databricks, tackling large-scale data and performance challenges. Ideal candidates have 5–7 years of data engineering...Remote workDatabricks
OpenKrill
New York, NY4 days ago - ...Ex is seeking a senior data engineer to lead end-to-end data engineering initiatives, including Azure Databricks development, PySpark migration, and scalable data integration for digital engagement. You will act as a single point of contact for client interactions,...Remote workDatabricks
Ex Services
New York, NY3 days ago - ...experienced Senior Data Software Engineer with strong hands-on expertise in Azure and the Spark ecosystem ,... ...adjacent technology (PySpark, Spark SQL, Scala Spark, Databricks, Synapse, etc.) rather... ...Responsibilities Lead the design, development, and optimization of scalable data...Databricks
- ...Machine Learning Engineers are... ...implementing data-driven solutions... ...for the Lead Machine Learning... ...(e.g., AWS, Azure, GCP).... ...a focus on optimization and scalability... ...leveraging SQL, Spark, NoSQL, and/... ...Snowflake, Databricks, Azure SQL,... ...employees to work remote when...Remote workDatabricksWork at officeVisa sponsorshipWork visaFlexible hours
Aimlroles
Charlotte, NC4 days ago - ...Lead Software EngineerIf you are looking... ...As a Lead Software Engineer at JPMorgan Chase... ...part of an agile data engineering team.... ...experience (AWS, Azure, or GCP)Ability to... ...platforms such as Databricks or SnowflakeDeep hands... ...on experience with Spark/PySpark and other big data...Databricks
hackajob
Houston, TX5 days ago - ...Senior Cloud Data Engineer / Data Architect Azure | Databricks | Data Lake | Synapse... ..., SQL, Python/PySpark, and related... .... Build and optimize batch and streaming... ...Databricks, Spark/PySpark, Azure... ...Location Primarily remote, with... ...to evolve as a leading provider of enterprise...Remote workDatabricksFull timeTemporary workWork at office1 day per week
TechTrend, Inc.
Reston, VA1 hour ago - Truelogic is seeking a Data Engineer with 5-7 years of hands-on experience to join our client’s team. You will design and optimize ETL pipelines, transform data, and ensure scalable... ..., fault-tolerant workflows using PySpark and Databricks. You will collaborate with cross-...Remote jobDatabricks
- ...Azure Databricks DeveloperLocation: Louisville... ...relocation or remote optionsJob... ...working with Azure Data Services,... ...develop, and optimize data pipelines... ...solutionsWrite advanced PySpark/Scala scripts... ...-tune Spark jobsImplement... ...experience in Data Engineering roles5-7+...Remote workDatabricksLong term contractContract workLocal areaRelocation
AceStack LLC
Louisville, KY5 days ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Lead Data Engineer (Azure Databricks, PySpark & Spark Optimization) - Canada (Remote). Be the first to apply!
