Lead Databricks Engineer with Pyspark & Python

Zortech Solutions

Role: Lead Databricks Engineer with Pyspark & Python Exp

Location: NYC, NY OR Iselin, NJ (Hybrid - 3 days' work from office)

Duration: 6+ Months

uery/Redshift experience is good to have
3+ years Python(numpy, pandas) coding experience
Experience with Snowflake utilities such as SnowSQL and SnowPipe - good to have
Experience in Data warehousing - OLTP, OLAP, Dimensions, Facts, and Data modeling.
Previous experience leading an enterprise-wide Cloud Data Platform migration with strong architectural and design skills
Capable of discussing enterprise level services independent of technology stack

Job Description:

Inperson interview is must, Expenses will be paid for in-person if not local

(Financial background is must)

Major Responsibilities:

Work on Finance data related to Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo
Design, develop, and deploy Databricks jobs to process and analyze large volumes of data.
Collaborate with data engineers and data scientists to understand data requirements and implement appropriate data processing pipelines.
Optimize Databricks jobs for performance and scalability to handle big data workloads.
Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks.
Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.
Demonstrated proficiency with Data Analytics, Data Insights
Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process
Azure Synapse/Bigquery/Redshift is good to have.
Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.

Skills:

5+ years' - Strong experience in Finance / Banking industry - Capital markets, investment banking - Collaterals, ETD, OTD, Settlement market, Cash product, Repo, Duos repo
10+ years - Enterprise Data Management
10+ years - SQL Server based development of large datasets
5+ years with Data Warehouse Architecture, hands-on experience with Databricks platform. Extensive experience in PySpark coding.
Azure Synapse/BigqExperience with Cloud based data architectures, messaging, and analytics
Superior communication skills
Cloud certification(s)
Any experience with Regulatory Reporting is a Plus

Education:

Minimally a BA degree within an engineering and/or computer science discipline
Master's degree strongly preferred

Apply

Vacancy posted 1 day ago

Similar jobs that could be interesting for youBased on the Lead Databricks Engineer with Pyspark & Python in Iselin, NJ vacancy

Lead Databricks Engineer with Pyspark & Python Exp
...Lead Databricks Engineer Location: NYC, NY OR Iselin, NJ (Hybrid – 3 days’ work... ...of data. Collaborate with data engineers and data scientists... .... Extensive experience in PySpark coding. ~ Azure Synapse/... ...is good to have ~3+ years Python(numpy, pandas) coding experience...
Databricks
Python
Work at office
Zortech Solutions
Iselin, NJ
1 day ago
Lead Databricks Engineer with Pyspark & Python Exp USA
...Lead Databricks Engineer Location: NYC, NY OR Iselin, NJ (Hybrid – 3 days’ work from office) Job... ...large volumes of data. Collaborate with data engineers and data scientists to... ...Databricks platform. Extensive experience in PySpark coding. ~ Experience with Cloud...
Databricks
Python
Work at office
Local area
Zortech Solutions
Iselin, NJ
1 day ago
Azure Databricks Architect with PySpark and Snowflake exp
...Role: Azure Databricks Architect with PySpark and Snowflake exp Location: Iselin... ...Snowflake ~3+ years Python (NumPy, Pandas) coding... ...~ Previous experience leading an enterprise-wide Cloud Data... ...Collaborate with data engineers and data scientists to understand...
Databricks
Python
Zortech Solutions
Iselin, NJ
1 day ago
Data Engineer
...Data Engineer Location: Mississauga, ON / Montreal... ...Snowflake SQL Python ADF PySpark Data Warehouse... ...Cloud Data engineer with a background in Python... ...deploy Spark program in databricks environment to process... ...Previous experience leading an enterprise-wide Cloud...
Databricks
Python
Sparktek
Iselin, NJ
2 days ago
Data Engineer
...Data Engineer Location: Hybrid 3 times a week too Iselin, NJ May Need to come onsite... ...on data engineers that have work with Java(spring Boot), Spark, Hadoop, and... ...CloudWatch) ETL/ELT Tools: AWS Glue, Python, PySpark, Databricks Programming Languages: Python, Java...
Databricks
Python
Work at office
Local area
RIT Solutions
Iselin, NJ
2 days ago
Databricks Architect/ Lead
...Role : Databricks Architect/ Lead with MDM exp Location : Iselin, NJ (Hybrid) Required : MDM Databricks Pyspark SQL Python Job Summary: We are seeking highly... ...: ~ Azure Data Engineer Associate or Databricks certified...
Databricks
Python
United IT Solutions
Iselin, NJ
4 days ago
Onsite senior developer/lead position
...Senior Onsite Developer/Lead Position Position: Consultant... ...years Location: Metropark, NJ with occasional visit to NYC... ...Core skills (STRONG): Azure Databricks (including Delta Lake, Medallion... ...Catalog), Azure Data Factory, Python, PySpark / SparkSQL, Data warehousing,...
Databricks
Python
Work at office
Local area
Software Technology Inc
Edison, NJ
3 days ago
Senior Data Engineer (MDM)
...Responsibilities: Lead the design,... ...the organization, with an emphasis on... ...leveraging modern data engineering tools and... ...data workflows using Python, Airflow, or Astronomer... ...using Kafka, Databricks, Snowflake, Azure... ...Expertise in Python, Pyspark, Kafka, microservices...
Databricks
Python
Flexible hours
Saxon Global
Iselin, NJ
21 hours ago
Technical Lead
1. Job Title : Client is looking for a Python/Pyspark Technical lead 2. Job Summary :Data Engineer: - Python / PySpark Engineer with 5-6 years of experience - Big Data ETL/ELT... ...REST API design ( Python - Flask) - Azure Databricks experience ( 1-2 years minimum) - PyTest...
Databricks
Python
Second job
Avani Technology Solutions
New Brunswick, NJ
22 days ago
Technical Delivery Manager
...strategy, experience design, and engineering, with a unique combination of agility, scale... ...-on Technical Delivery Managerto lead the end-to-end delivery of a... ...* Provide hands-on guidance in Databricks, Azure OneLake, PySpark, Python, and .NET * Oversee data engineering...
Databricks
Python
Orion Innovation
Edison, NJ
21 hours ago
Data Integration Operations Manager (VP Level)
$170k
...integration processes, and partner with stakeholders to drive... ...environments Strong SQL and Python development skills Expected... ...platforms such as Python and Databricks Ability to collaborate... ...Oracle/SQL Server, SSIS, Python, Pyspark, Databricks, Azure, ADF,...
Databricks
Python
Local area
Shulman Fleming & Partners
Iselin, NJ
9 days ago
ETL Developer
...• Migrate existing SSIS ETL scripts to Python; develop new ETL scripts • Support existing... ...using cloud-based • Work closely with existing senior integration staff to flush... ...pipelines from multiple sources using Azure Databricks. • Direct experience of building data piplines...
Databricks
Python
Full time
Remote work
2 days per week
Seven Seven Software
Iselin, NJ
4 days ago
Azure Data Tech Lead
...Azure Data Tech Lead Location: Alpharetta, Georgia... ...– Yrs of Exp – Azure Databricks Skill 2 – Yrs of... ...Skill 2 – Yrs of Exp – Python Core Skills:... ...experienced Senior/Lead Data Engineer with 8+ years of expertise... ...Spark with Python (PySpark). • Strong command over...
Databricks
Python
Cardinal Integrated
Maplewood, NJ
21 hours ago
Cloud Data Engineer
..., AWS, or Google Cloud. Data Engineering : Strong understanding of data warehousing... .... Programming: Experience with programming languages like Python, Java, or Scala. Cloud Data... ...experience with Cloud Data Platform like Databricks (preferred), Snowflake, Redshift,...
Databricks
Python
3B Staffing LLC
Iselin, NJ
2 days ago
Airflow Orchestration - Solutions Architect
...Cloud Orchestration, will work with a small group of... ...platform. In this role, he will engineer a wide range of use cases where... ...use cases. Proficiency in Python. Knowledge of Azure cloud... ...Experience Snowflake and Databricks experience. Experience working...
Databricks
Python
United IT Solutions
Iselin, NJ
1 day ago
Data Platform engineer - Azure admin
...Data Platform Engineer - Azure Admin Location: Iselin, NJ (Hybrid) Contract... .... Experienced working on Databricks workspaces, workload optimisation,... ...automating tasks through scripting (Bash, Python, Powershell). Experience with configuration management (e.g. Puppet...
Databricks
Python
Contract work
United IT
Iselin, NJ
2 days ago
AWS Databricks Platform Administrator
...Job Title - Databricks Platform Administrator / Engineer Location - Hybrid Duration - 12+ Months... ...Platform Administrator / Engineer with proven experience supporting modern... ...in scripting and automation using Python, Bash, PowerShell , or Terraform...
Databricks
Python
3B Staffing LLC
Colonia, NJ
1 day ago
Azure Databricks Engineer
...Azure Databricks Engineer Job Location: Berkeley Heights, NJ, Omaha, NE, or Frisco, TX (100%... ...the scrum team. Interacts effectively with clients and leadership, and can adapt... ...Explorer, Azure Event Hubs, Spark Pools, Python, PySpark, SOL, Azure Landing Zone, Azure...
Databricks
Python
Local area
Relocation
Veracity
Berkeley Heights, NJ
22 days ago
Marketing Operations Analyst
...initiatives, and ensuring compliance with privacy regulations such as... ...teams such as IT and engineering to integrate data from various... ...and analysis. Comfortable with Python. Experience working within data platforms like Databricks/Snowflake, and analytics modeling...
Databricks
Python
Full time
FocusKPI
Edison, NJ
4 days ago
Lead Software Engineer - Finance / FinTech
2Bridge has been engaged in the search for multiple full-time, direct-hire Lead Full Stack Software engineers with a background in Python, React, and Typescript on AWS with our financial client located near Colonia, NJ. They offer a comprehensive package that includes...
Python
Full time
2Bridge Partners
Woodbridge, NJ
4 days ago
Sr. Data Scientist
...Point, PA. Qualifications: Candidates with a degree in computer science or related... ...Experience with programming in Python and deployment on Posit Experience with... ...with data pipeline tools such as Dataiku or Databricks Experience with relational databases and...
Databricks
Python
3 days per week
AA2IT
Rahway, NJ
2 days ago
Lead Specialty Software Engineer
$159k - $305k
...this role: Wells Fargo is seeking a Lead Specialty Software Engineer in technology as part of Commercial... ...platform, balancing low-latency components with scalable services to support strategy... ...programming languages such as Java, Python or C++ 4+ years of deep understanding...
Python
Work experience placement
Relocation package
Wells Fargo
Iselin, NJ
4 days ago
Lead Gen AI Engineer
...contingency plans to address risks. (4.) To lead and empower a dynamic development team... ...: GenAIMachine Learning Prompt Engineering Python scripting Azure AI Services... ...Problem Management) Good coordination with multiple stakeholders, vendors and service...
Python
Omni Inclusive
Rahway, NJ
2 days ago
Lead Software Engineer - Banking
$110k - $180k
...Summary : Join a leading wealth management firm dedicated to delivering... ...clients. As a Lead Software Engineer, you will play a pivotal role... .... Collaborate with product managers and stakeholders... ...as TypeScript, JavaScript, and Python to deliver robust software solutions...
Python
Temporary work
Work at office
Remote work
Bessemer Trust
Woodbridge, NJ
4 days ago
Databricks Data Engineer - Manager - Consulting - Miami
$125.5k - $230.2k
...in to shape your future with confidence. We’ll... ...Decision Science – Data Engineering – Manager We are... ...of Data Engineering to lead our team in designing and... ...with a strong focus on Databricks. The ideal candidate... ...skills in languages such as Python, Scala, or SQL....
Databricks
Python
Summer holiday
Flexible hours
EY
Iselin, NJ
3 days ago
AI Solution Architect
...Architects who will serve as lead developers on the... ...degree in Computer Science, Engineering, Data Science, or... ...development experience with 4+ years in AI/ML architecture... ...mining Azure Databricks: Data engineering and... ...Programming Languages: Python, C#, JavaScript/TypeScript...
Databricks
Python
Remote work
Work visa
EisnerAmper
Iselin, NJ
2 days ago
Principal Data Analyst- Capital Markets
$140k - $170k
...Markets Data & Analytics lead for Citizens Bank. This... ...intersection of data engineering, applied analytics, and... ...and partnering closely with product, engineering, and... ...) ~ Strong SQL and Python skills; experience with... ...platforms (Snowflake, AWS, Databricks). ~ Demonstrated...
Databricks
Python
Local area
Remote work
Monday to Friday
Flexible hours
Citizens Financial Group, Inc.
Iselin, NJ
2 days ago
Databricks Data Engineer - Senior - Consulting - Location Open
$106.9k - $176.5k
...in to shape your future with confidence. We’ll... ...Decision Science – Data Engineering – Senior We are... ...engineering, specifically Databricks. The ideal candidate... ...business objectives. Lead end-to-end data pipeline... ...skills in languages such as Python, Scala, or SQL....
Databricks
Python
Summer holiday
Flexible hours
EY
Iselin, NJ
1 day ago
AI Platform Architect
...to the AI Development Lead, this is a high-impact,... ...intersection of architecture, AI engineering, and strategic... ...boundaries-partnering with IT infrastructure teams... ...Azure SQL, Azure Synapse/Databricks for analytics and data... ...Programming Languages: Python, C#, TypeScript, SQL, Bicep...
Databricks
Python
Local area
Work visa
EisnerAmper
Iselin, NJ
4 days ago
Data Architect - Power & Utilities - Manager- Consulting - Location OPEN
$125.5k - $230.2k
...in to shape your future with confidence. We’ll... ...power systems / electrical engineering background and having... ...the opportunity to lead and develop your skill... ...data engineering, such as Python, Java, or Scala. ~ Ability... ...data platforms like Databricks. Experience in...
Databricks
Python
Summer holiday
Flexible hours
EY
Iselin, NJ
21 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Lead Databricks Engineer with Pyspark & Python. Be the first to apply!