Senior Data Engineer, Databricks

Design, build, and test end-to-end data pipelines, including streaming, events, and batch data ingestion, on cloud-based infrastructure using Azure and Cosmos DB Develop and implement ETL processes for data ingestion, transformation, and loading into data lakes Work extensively on Databricks and data warehousing concepts Design and develop custom high-throughput and configurable frameworks and libraries Drive change through collaboration, influence, and demonstration of POCs Manage all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation Work collaboratively within an agile project team Follow best practices and coding standards Grow personal skillset Requirements Extensive technical, people and process skillsets Strong background in application architecture and development Very good understanding of leading IT technologies Experience in micro-services architecture and understanding of Cloud Computing; Azure preferred Proficient working knowledge of Databricks, Delta Lake, and Apache Spark Hands-on development experience in Databricks SQL and Scala Strong understanding of the Databricks platform, including clusters, jobs, and other resources Ability to monitor and troubleshoot data pipelines Experience implementing data quality checks and validations Experience performing data analysis and data exploration Strong hands-on experience troubleshooting DevOps pipelines, including ADO and Harness Experience in a multi-developer environment using version control such as Git Experience handling semi-structured data including Avro, JSON, and XML Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer Bachelor’s Degree level qualification in a computer or IT-related subject 10+ years of overall Big Data data pipeline experience 5+ years of hands-on Databricks experience 5+ years of cloud-based development experience, including Azure Services, Azure DevOps, Kubernetes, and Docker Experience with Snowflake is a plus Effective professional written and oral communication Team player with a positive attitude, enthusiasm, initiative, and self-motivation Ability to multitask; energetic, fast learner, and problem solver Experience working in the financial industry Experience with agile development methodology Core Competencies Demonstrates expertise in designing and implementing end-to-end data pipelines using Azure, Databricks, and cloud-based infrastructure, with a strong focus on ETL processes and data quality. Proficient in application architecture, micro-services, and agile methodologies, with a proven ability to collaborate effectively within teams. Highest-signal resume keywords Azure Cloud Computing Databricks SQL ETL Processes Data Pipeline Monitoring Agile Development Methodology ATS Optimization Keywords Hard Skills Data Pipeline Development Databricks Apache Spark Scala SQL Data Quality Checks Shell Scripting UNIX Version Control (Git) Data Analysis Soft Skills Effective Communication Team Collaboration Problem Solving Self-Motivation Positive Attitude Certifications & Qualifications Bachelor’s Degree in Computer or IT-related Subject Industry Keywords Big Data Financial Industry Agile Methodology Micro-Services Architecture Tools & Technologies Azure DevOps Kubernetes Docker Cosmos DB Snowflake TOAD Delta Lake ADO Harness #J-18808-Ljbffr Jobtailor