Azure Databricks Lead

HCLTech is looking for a highly talented and self-motivated Azure Databricks Technical Lead to join it in advancing the technological world through innovation and creativity. Job ID: 156897 Position Type: Full-time (Hybrid) Location: New York, NY Role/Responsibilities Design, develop, and maintain scalable data engineering solutions using Azure Databricks, Azure Data Factory (ADF), and Azure Data Lake Storage (ADLS Gen2). Build and optimize batch and real-time data processing pipelines using PySpark and Spark SQL. Develop reusable, maintainable, and high-performance Python applications following OOP principles and advanced design patterns. Implement Delta Lake-based data solutions to support enterprise data lakehouse architectures. Design and optimize distributed data processing frameworks for large-scale datasets. Develop and maintain ETL/ELT processes ensuring data quality, reliability, and performance. Collaborate with business stakeholders, architects, and cross-functional teams to gather requirements and deliver scalable cloud data solutions. Monitor, troubleshoot, and optimize data pipelines and workloads running on Azure platforms. Support data governance, security, and compliance initiatives across enterprise data environments. Contribute to architecture discussions, code reviews, and best-practice implementation. Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field. Strong hands-on experience with: Expert-level proficiency in Python programming. Strong understanding of Object-Oriented Programming (OOP) concepts and advanced Python design patterns. Extensive experience with: PySpark Spark SQL SQL Strong knowledge of: Data Lakehouse Architecture Distributed Computing Concepts Experience building scalable cloud-native data processing solutions. Strong analytical, troubleshooting, and performance optimization skills. Excellent communication and collaboration skills. Level of Education Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, Data Analytics, or a related technical discipline. Equivalent professional experience may be considered. Desired Qualifications Experience with CI/CD pipelines and DevOps practices for data engineering. Exposure to Azure Synapse Analytics, Azure Fabric, or other Azure analytics services. Experience implementing enterprise-scale data lakehouse solutions. Knowledge of data governance, data quality, and security frameworks. Experience working in Agile/Scrum environments. Pay and Benefits Pay Range Minimum: $127,000 per year Pay Range Maximum: $180,000 per year HCLTech is an equal opportunity employer, committed to providing equal employment opportunities to all applicants and employees regardless of race, religion, sex, color, age, national origin, pregnancy, sexual orientation, physical disability or genetic information, military or veteran status, or any other protected classification, in accordance with federal, state, and/or local law. Should any applicant have concerns about discrimination in the hiring process, they should provide a detailed report of those concerns to View email address on click.appcast.io for investigation. A candidate's pay within the range will depend on their skills, experience, education, and other factors permitted by law. This role may also be eligible for performance-based bonuses subject to company policies. In addition, this role is eligible for the following benefits subject to company policies: medical, dental, vision, pharmacy, life, accidental death & dismemberment, and disability insurance; employee assistance program; 401(k) retirement plan; 10 days of paid time off per year (some positions are eligible for need-based leave with no designated number of leave days per year); and 10 paid holidays per year. How You'll Grow At HCLTech, we offer continuous opportunities for you to find your spark and grow with us. We want you to be happy and satisfied with your role and to really learn what type of work sparks your brilliance the best. Throughout your time with us, we offer transparent communication with senior-level employees, learning and career development programs at every level, and opportunities to experiment in different roles or even pivot industries. We believe that you should be in control of your career with unlimited opportunities to find the role that fits you best. #J-18808-Ljbffr HCLTech