Senior Data Engineer

Overview

Hudson Valley Credit Union is currently recruiting for the position of Senior Data Engineer. Primary Functions: Design, build, and optimize our data infrastructure, leveraging the power and scalability of Databricks within our Azure environment. Maintain a deep expertise in Databricks and its ecosystem in order to build high-performance data pipelines, ensure data quality, and enable advanced analytics and machine learning initiatives.


Responsibilities

• Design, develop, and maintain robust and scalable data pipelines primarily using Databricks, leveraging Spark (PySpark or Scala), Delta Lake, and Databricks Workflows
• Build and optimize data lakehouse solutions on Azure Data Lake Storage Gen2 integrated with Databricks
• Implement and enforce data quality frameworks and data governance policies within Databricks using tools and features like Delta Lake's ACID properties and data validation techniques
• Identify and resolve performance bottlenecks in Databricks-based data pipelines and Spark applications
• Collaborate closely with analysts to understand their data requirements for advanced analytics, providing them with curated and performant datasets within Databricks
• Automate data ingestion, transformation, and deployment processes, potentially integrating with CI/CD tools and Databricks MLOps
• Provide technical leadership and mentorship to junior data engineers on Databricks best practices and performance tuning
• Document Databricks notebooks, data pipelines, data models, and configurations
• Ensure data security and compliance within the Databricks environment, adhering to relevant security standards and best practices
• Troubleshoot and resolve complex data-related issues within the Databricks ecosystem
• Work independently and contribute to immediate and cross-functional teams
• Participate in design discussions
• Analyze problem statements, break down problems, and provide solutions
• Follow an agile-based approach to deliver data products
• Take ownership of large tasks, delivering on time, and mentoring team members
• Oversee the maintenance and enforcement of SDLC and ITSM processes, change management procedures, departmental standards, style requirements and all other procedures necessary for data and system integrity
• Adhere to all Credit Union policies, procedures, and regulatory agency requirements. Participate in all required and recommended training and development including, but not limited to, Bank Secrecy Act training (BSA) and demonstrate attained knowledge. Participate on Credit Union teams, projects and strategic initiatives when the opportunity arises. Perform additional duties and special projects as assigned.
• Embrace and apply HVCU's guiding principles to all activities and responsibilities. This includes the Credit Union's Mission, Vision, Core Values, Employee and Member Value Propositions, Sales and Service Model, and commitment to Lean Six Sigma practices. Support the Credit Union's initiatives by demonstrating teamwork and professionalism.
• Responsible for regular and predictable attendance including punctuality.

Qualifications

• Bachelor's Degree or equivalent in Computer Science or related field required.
• Master's Degree or equivalent in Computer Science or related field preferred.
• License/Certification Databrick Data Engineer Associate required.
• License/Certification Databrick Data Engineer Professional preferred.
• License/Certification Mircrosoft Azure Data Engineer preferred.
• Minimum 5 years of experience as Data Engineer required.
• Minimum 5 years of experience Data Wharehousing, Data Lakes and big date platforms required.
• Minimum 5 years Python experience required.
• Minimum 3 years Databricks experience required.

Click here to view full job description

Why Choose HVCU?

Join a team where your growth matters. At Hudson Valley Credit Union, we offer a supportive, community-focused culture, competitive benefits, and opportunities to grow your career while making a meaningful impact. Come be part of a Great Place to Work!


Employee Perks:

• Base & Variable Pay Structures

• Paid Personal, Holiday & Volunteer Time Off

• Professional Development & Training

• Student Loan Repayment & Tuition Reimbursement Programs

• Medical, Dental & Vision Coverage

• 401(k) With Employer Match & Non-Elective Employer Contribution

• Life, Short-term & Long-term Disability Insurances

• Discounted Loan Rates & Fees

Hudson Valley Federal Credit Union is an AA/EEO employer committed to equal opportunity and employee diversity. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, age, sexual orientation, gender identity, national origin, disability, military or veteran status, genetic information, marital or familial status, domestic violence victim status, or any other characteristic protected by law.