Data Engineer - Databricks

We are seeking a Data Engineer with strong Databricks expertise to modernize and scale our Business Intelligence (BI) capabilities. This role will design and build data pipelines, deploy machine learning solutions, and operationalize intelligent analytics to drive decision-making across the organization. The ideal candidate blends data engineering best practices with applied machine learning, MLOps, and AI.


Key Responsibilities


Project Responsibility: End-to-end data pipelines and integrations


Technical Competencies:

• Advanced SQL optimization and complex query design
• Kafka streaming applications and connector development
• Databricks workflow development with medallion architecture
• Data governance implementation and compliance
• Performance tuning for large-scale data processing
• Data security and privacy best practices
• Apache NiFi pipeline development for invoice and PO processing
• Integration with purpose-built data stores (Druid, MongoDB, OpenSearch, Postgres)
• Build and maintain end-to-end ML pipelines for training, deployment, and monitoring of models.
• Design and optimize data architectures for large-scale ML workloads
• Explore and implement LLM-based solutions, RAG architectures, and generative AI for business use cases.

Soft Skills:

• Cross-functional collaboration with product and engineering teams
• Technical mentoring for junior data engineers
• Analytical thinking for complex data problems
• Stakeholder communication for data requirements
• Process improvement and efficiency focus
• Quality mindset for data accuracy and reliability Vendor Management:
• Direct communication with data platform vendors
• Evaluates vendor tools for specific data use cases
• Provides technical feedback on vendor product roadmaps
• Coordinates with vendors for data integration projects

Qualifications:


Bachelor's/Master's in Computer Science, Data Engineering, Statistics, or related field.


5 years in data engineering; 2 years applying ML in production.


Determining compensation for this role (and others) at Vaco/Highspring depends upon a wide array of factors including but not limited to the individual's skill sets, experience and training, licensure and certifications, office location and other geographic considerations, as well as other business and organizational needs. With that said, as required by local law in geographies that require salary range disclosure, Vaco/Highspring notes the salary range for the role is noted in this job posting. The individual may also be eligible for discretionary bonuses, and can participate in medical, dental, and vision benefits as well as the company's 401(k) retirement plan. Additional disclaimer: Unless otherwise noted in the job description, the position Vaco/Highspring is filing for is occupied. Please note, however, that Vaco/Highspring is regularly asked to provide talent to other organizations. By submitting to this position, you are agreeing to be included in our talent pool for future hiring for similarly qualified positions. Submissions to this position are subject to the use of AI to perform preliminary candidate screenings, focused on ensuring minimum job requirements noted in the position are satisfied. Further assessment of candidates beyond this initial phase within Vaco/Highspring will be otherwise assessed by recruiters and hiring managers. Vaco/Highspring does not have knowledge of the tools used by its clients in making final hiring decisions and cannot opine on their use of AI products.


EEO Notice


Vaco by Highspring is an Equal Opportunity Employer and does not discriminate against any employee or applicant for employment because of race (including but not limited to traits historically associated with race such as hair texture and hair style), color, sex (includes pregnancy or related conditions), religion or creed, national origin, citizenship, age, disability, status as a veteran, union membership, ethnicity, gender, gender identity, gender expression, sexual orientation, marital status, political affiliation, or any other protected characteristics as required by federal, state or local law.


Vaco by Highspring and its parents, affiliates, and subsidiaries are committed to the full inclusion of all qualified individuals. As part of this commitment, Vaco by Highspring and its parents, affiliates, and subsidiaries will ensure that persons with disabilities are provided reasonable accommodations. If reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, please contact View email address on click.appcast.io .


Vaco by Highspring also wants all applicants to know their rights that workplace discrimination is illegal .

Representation Notice


By submitting to this position, you agree that you will be giving Vaco by Highspring the exclusive right to present your as a candidate for the foregoing employment opportunity. You further agree that you have represented information about yourself accurately and have not affirmatively misrepresented your qualifications. You also agree to maintain as confidential, to the fullest extent permitted by law, any information you learn from Vaco by Highspring about the position and you will limit disclosure of information about the position only to the extent necessary to perform any obligations in furtherance of your application. In exchange, Vaco by Highspring agrees to exercise reasonable efforts to represent you through all solicitation, job screening and resume dispersal.

For residents of Ontario, Canada: Based on Highspring's discussions with its Client, Highspring's understanding is that this position for employment is a current vacancy (either through Highspring as a contractor or with the client directly).

Privacy Notice


Vaco by Highspring and its parents, affiliates, and subsidiaries ("we," "our," or "Vaco by Highspring") respects your privacy and are committed to providing transparent notice of our policies.

• California residents may access Vaco by Highspring HR Notice at Collection for California Applicants and Employees here .
• Virginia residents may access our state specific policies here .
• Residents of all other states may access our policies here .
• Canadian residents may access our policies in English here and in French here .
• Residents of countries governed by GDPR may access our policies here .

Additionally, submissions to this position are subject to the use of AI to perform preliminary candidate screenings, focused on ensuring minimum job requirements noted in the position are satisfied. More details about Vaco by Highspring's use of AI can be found here ( ). Further assessment of candidates beyond this initial phase will be conducted by recruiters and hiring managers. Vaco by Highspring does not know and cannot opine on if its client's use of AI products in hiring.

Pay Transparency Notice


Determining compensation for this role (and others) at Vaco by Highspring depends upon a wide array of factors including but not limited to:

• the individual's skill sets, experience and training;
• licensure and certification requirements;
• office location and other geographic considerations;
• other business and organizational needs.

With that said, as required by local law, Vaco by Highspring believes that the following salary range referenced above reasonably estimates the base compensation for an individual hired into this position in geographies that require salary range disclosure. The individual may also be eligible for discretionary bonuses.