Databricks Data Engineer
MDMS Recruiting
6 month contract opportunity in NYC | W2 Only We are seeking a skilled and experienced professional to join our team. You will be responsible for designing, developing, and implementing data engineering solutions using the Databricks platform to support data ingestion, transformation, performance reporting, analytics, sales incentive compensation processing, and enterprise data pipeline processes. Responsibilities
- Collaborate with business stakeholders to understand their data requirements and translate them into scalable technical solutions.
- Design and develop Databricks-based data pipelines, including data models, Delta Lake tables, and dashboards/notebooks.
- Responsible for the overall Databricks architecture to ensure new pipelines and enhancements to existing workflows are designed in an integrated and optimal manner.
- Maintain and optimize all current Databricks workspaces and pipelines; capture and deliver requirements for new or improved functionality to achieve further automation and integration of data and processes.
- Design and implement data pipelines to support Sales Incentive Compensation (SIC) processes, including quota management, attainment calculations, commission processing, and payout reporting.
- Partner with Sales Operations and Finance teams to understand incentive plan rules and translate them into accurate, auditable data transformations and models.
- Perform testing and debugging of data pipelines and notebooks to identify and resolve issues and ensure optimal performance.
- Provide technical support and troubleshooting for Databricks-based solutions, working closely with end-users and analysts to address their needs and resolve any data-related problems.
- Stay up to date with Databricks features and enhancements, and proactively identify opportunities to leverage them to improve data quality, performance, and analytics capabilities.
- Collaborate with cross-functional teams, including finance, IT, and business intelligence teams, to ensure seamless integration and alignment of Databricks with other systems and processes.
- Document technical specifications, data models, pipeline configurations, and development processes to ensure knowledge transfer and maintain system documentation.
- At least 6 years of experience in data engineering, with a minimum of 2-3 years hands-on experience building and managing Databricks solutions.
- Extensive experience with Databricks platform components: Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
- Strong proficiency in Python and/or Scala for data engineering; SQL expertise required.
- Strong experience with core data engineering practices including data ingestion, transformation (ETL/ELT), data modeling, and pipeline orchestration.
- Functional and technical experience with Sales Incentive Compensation (SIC) - including incentive plan design, quota allocation, attainment tracking, commission calculations, and payout processing.
- Familiarity with SIC platforms and experience integrating them with data engineering platforms is a strong plus.
- Ability to understand complex incentive compensation rules and model them accurately in data pipelines and transformation logic.
- Experience integrating Databricks with cloud platforms and data sources such as Azure Data Factory, Event Hubs, Kafka, or equivalent.
- Passion and ability to create holistic, end-to-end, integrated data solutions.
- Stakeholder management skills; ability to challenge constructively and translate complex business requirements into scalable data architectures.
- Strong analytical and problem-solving skills, with the ability to understand complex data requirements and implement reliable, efficient solutions.
- Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams and stakeholders.
- Must work well in a team environment while being a resourceful, independent self-starter able to work effectively with minimal direction.
- Skilled at documenting complex data architectures and pipelines in a professional manner.
- Ability to work independently and manage multiple tasks and priorities in a fast-paced environment.
- Passion for data quality, performance optimization, and delivering excellent end-user data experiences.
- Databricks Certified Data Engineer Associate or Professional certification is a plus.
Vacancy posted 4 days ago
Similar jobs that could be interesting for youBased on the Databricks Data Engineer in New York, NY vacancy
- ...Territories, New York, HKPosted: 2025-11-25Unlock your potential as a Data Engineer with a focus on innovation and impact in Venus's Team. This... ...the opportunity to work with cutting-edge technologies like DataBricks, aiming to enhance data architectures and drive analytics...DatabricksPermanent employment
- Job ID: 25546234Reference Number: 25-00476Title: Data EngineerLocation: New York, NYPosted Date: 2025-05-19Contact: Rishabh PandeyContact... ...data lakehouse platforms (preferably AWS data lake services or Databricks, alternatively Hadoop), atleast one relational data stores (...Databricks
$100k - $150k
...New York, United States Permanent Data Catalog EngineerNew York, NY - Hybrid (3... ...) My client is seeking a Data Catalog Engineer to join a growing Data Governance team focused... ...catalog platforms like Atlan, Collibra, Databricks Unity Catalog, Alation, Informatica or...DatabricksPermanent employmentWork at office3 days per week- ...3 Must Have Skills:Informatica workflow design and implementation requiredAWS data engineering (Glue, Lambda, EventBridge Cradle, S3) highly preferredFormer Informatica experience, Databricks, SnowflakeNice to Haves:Apache AirflowSomeone who is fun and exciting with lots...DatabricksWork at office
$100 - $130 per hour
A financial firm is looking for a Data Engineer - Databricks to join their team. This role is remote. Pay: $100-130/hrUS Citizens or GC Holders OnlyNo c2c - W2 OnlyQualifications:A Bachelor's Degree in Computer Science, Management Information Systems, Computer Engineering...DatabricksRemote work- ...Data Engineer Key skills are very strong SQL capability, data pipelines management, Python, and Airflow. Location is New York, mandatory... ...to develop DAGS. Expert in data warehouses like BigQuery, Databricks Deltalakehouse and how to programmatically ingest, cleanse,...Databricks3 days per week
Yochana
New York, NY3 days ago - ...Required Skills and Experience: Proven experience as a Data Engineer or in a similar role with a focus on Snowflake, Python, Pyspark... ...Preferred Qualifications: Good Knowledge in Azure and Databricks in highly preferred. Knowledge of Apache Kafka or other...Databricks
Saxon Global
New York, NY5 days ago - ...Job Title- Data Engineer Location- New York, NY 10112 Reporting Type- Onsite Duration: 10 months Summary... ...databases. Structured and unstructured data sets. Loading of data into S3 as well as tabular schemas. Databricks experience preferred.Databricks
campus4tech
New York, NY2 days ago - ...Enterprise Data And Analytics Platform Architect Candidate would be working on the... ...Experience with software and cloud engineering and architecture in building enterprise... ...solutions using Azure Data lake, Azure Databricks and PowerBI Experience creating data...Databricks
Samprasoft
New York, NY2 days ago - ...Job Title Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Lake. Integrate data from various sources into the Databricks platform. Implement data integration and ETL processes using Azure Data Factory. Develop and...Databricks
Omni Inclusive
New York, NY2 days ago - ...About the job DataBricks Data Engineer DataBricks Data Engineer requires: ~ Must have functional and technical experience with Sales Incentive Compensation (SIC) including incentive plan design, quota allocation, attainment tracking, commission calculations...Databricks
Global Channel Management
New York, NY3 days ago - ...successful and growing investment management firm seeking a Senior Data Engineer to help transform and modernize its enterprise data platform.... ...QualificationsExperience with Azure, Azure Data Factory, Databricks, or Synapse.Experience migrating data platforms from SQL...DatabricksWork at officeImmediate startWork from home
- ...Data Engineer Location: NJ / NY onsite USA (local to NJ or NY) Must have: (AWS, Azure, or GCP), and/or on-prem (MS SQL, SSIS, Oracle) and (RedShift, Snowflake, Databricks, Informatica, SSAS etc.) Requirements: ~ Bachelor's degree in Computer Science, Data...DatabricksLocal area
Damco
New York, NY4 days ago - QUALIFICATIONSDegree in Computer Science, Business Analytics, Engineering, Mathematics, or related field2+ years of professional experience in data engineering, software engineering, or... ..., and Delta LakePractical experience with Databricks, AWS Glue, and transformation frameworks...DatabricksApprenticeshipEasy work
$125k - $145k
...we're buildingOur team runs a range of data and AI projects across the GPS business.... ...forThis is a hybrid role that spans data engineering, AI product enablement, and platform architecture... ...technologies (e.g., Snowflake, Databricks, Delta Lake) alongside distributed storage...DatabricksFull timeWork at officeFlexible hoursDay shift$190k - $220k
...role and will be assessed during the interview process.The Data Engineering team builds tools and systems that make Gusto's data consistent... ...on cloud platforms like Snowflake, Redshift, BigQuery, or Databricks.Experience implementing CI/CD pipelines, automated testing,...DatabricksFull timeWork at officeLocal area2 days per week3 days per week$110k - $130k
.... About the Role:At The New York Times, data powers decisions across the entire company... ...possible, and we're looking for a Data Engineer to help us build them. You'll own and... ...Cloud Data Warehouse (BigQuery, Snowflake, DataBricks)Familiarity with foundational cloud...DatabricksFull timeLocal areaFlexible hours- ...Your Role The Data Services team is responsible for technical design and end-to... ...data products for the enterprise. The Data Engineer,Senior will report to the Sr Manager,... ...data stack tools such as dbt, Snowflake, Databricks, Airflow, or Tidal Solid knowledge...Databricks
Blue Shield of CA
New York, NY5 days ago - ...WithAt Slalom, we co-create custom software, data, and cloud products with clients who are... ...true change.Our Data Architecture & Engineering team sees data as the foundation for... ...Data & Streaming — Apache Spark, Presto, Databricks, Amazon Kinesis, Apache Kafka* Workflow...DatabricksTemporary workLocal area
$147k - $181k
...shape a brighter way forward. About JLL and Data PlatformJLL is a leading professional... ....What this job involvesAs a Senior Data Engineer on the Data Consumption team, you will... ...Preferred QualificationsExperience with Databricks, Unity Catalog, or similar data catalog...DatabricksFull timeContract workLocal areaRemote work- ...Data Engineer Design, build, and optimize scalable ELT/ETL pipelines ingesting banking and treasury data from Kyriba into Snowflake and Databricks. Develop and own canonical data models and schemas for cash positions, bank transactions, intercompany settlements, and...Databricks
TPI Global
New York, NY2 days ago $116.2k - $229.1k
Position Summary Join Deloitte’s AI & Engineering practice and help organizations transform enterprise technology platforms, modernize data environments, and unlock value through innovation. As a Databricks Engineer in our AI & Data practice, you will design, build...DatabricksLocal areaVisa sponsorship$180k - $220k
...unlocks the magic of frictionless experiences.We’re looking for a Data Engineer II to help us build the next generation of products which... ..., Luigi, etcBig data tools like Spark, Kafka, Snowflake, Databricks, etcCollaboration, integration, and deployment tools like Github...DatabricksCasual workWork at officeFlexible hours$137.4k - $161.7k
...you ready to make an impact?West Monroe is seeking a Senior Data Engineer to join our Data & AI practice. In this role, you will design... ...and business intelligence. Working with technologies such as Databricks, Snowflake, and Microsoft Fabric, you will help clients modernize...DatabricksLocal areaImmediate startFlexible hours- ...Data Engineer Work location: New York City, NY (Hybrid, 3 days in a week) In-person Interview is required Client: Mizuho Bank Visa:... ...Advanced proficiency in Python, SQL Server, Snowflake, Azure Databricks, and PySpark. Strong understanding of relational databases...Databricks3 days per week
Argyle Infotech
New York, NY4 days ago $110k - $190k
Senior Data Management Professional - Data Engineering - Entities Location New York Business Area Data Ref # 10052582 Description... ...services such as S3, EMR, Glue, ADLS, Data Factory, or Databricks.• Experience implementing data observability tools...DatabricksTemporary workFor contractorsWork experience placement$190.8k - $238.8k
..., Raleigh, London, and Amsterdam.Making data-driven decisions is key to Plaid's culture... ...datasets and tooling to teams across engineering, product, and business and help them explore... ...and data lakes; Redshift, Snowflake, Databricks.You have experience building and maintaining...DatabricksFull timeWork experience placementLocal areaFlexible hours$200k - $250k
...RolePolymarket is looking for a Senior Analytics Engineer to own the modeling layer that sits between our raw data and the decisions the business makes with it.... ...solutions, and a modern cloud warehouse such as Databricks, Snowflake, BigQuery, or ClickHouseStrong instincts...Databricks- ...Instacart, Atlassian, Synopsys, Stripe, Databricks, and Waymo use Pave, transforming every... ...Venture Partners, and Craft Ventures.The Data Team @ PaveAs part of the Data team at Pave... ..., product managers and software engineers to translate product needs into insights...DatabricksFlexible hours3 days per week
- ...Databricks Data Engineer Location: New York, NY (Hybrid – 4 days onsite per week - Tues/Wed anchor days) Job Type: Contract (6 Months) Schedule: Monday – Friday, 8:00 AM – 5:00 PM About the Opportunity We are seeking an experienced Databricks Data Engineer to...DatabricksContract workMonday to Friday
ThreePDS Inc.
New York, NY18 hours ago
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Databricks Data Engineer. Be the first to apply!
Related searches
- data platform engineer New York, NY
- junior data engineer New York, NY
- data integration engineer New York, NY
- bi data engineer New York, NY
- aws data engineer New York, NY
- data engineer intern New York, NY
- data science developer New York, NY
- sr information security engineer New York, NY
- data engineer graduate New York, NY
- senior data engineer New York, NY