Azure DataBricks Developer

Hello,


Greetings!!!


This is Abhishek from Jconnect INC . Below is the requirement with my client. Please let me know if you are available for this role.

Title: Azure DataBricks Developer


Location: Louisville,KY/ Chicago,IL/ Dallas,TX / Arlington,VA


Duration: Fulltime

JOB DESCRIPTION:
• Design, develop, and maintain cloud native data engineering solutions using Azure Databricks.
• Build and manage PySpark notebooks to process large scale structured and semi structured datasets.
• Design, create, and maintain Delta Lake tables, ensuring data reliability, ACID transactions, and schema enforcement.
• Develop scalable data workflows and pipelines using Databricks notebooks and orchestration patterns.
• Optimize performance of Spark jobs, including tuning partitions, memory usage, caching strategies, and query execution.
• Work extensively with PySpark and Spark SQL, choosing the appropriate approach based on use case and performance needs.
• Support cloud data migration initiatives, migrating data pipelines from on prem or legacy platforms to Azure Databricks.
• Integrate Databricks with upstream and downstream systems (e.g., data sources, storage layers, reporting tools).
• Ensure data pipelines are robust, reusable, and maintainable, following enterprise data engineering best practices.
• Implement error handling, logging, monitoring, and recovery strategies for production grade data pipelines.
• Collaborate with data architects, analysts, and downstream consumers to understand data requirements.
• Perform debugging and root cause analysis for data quality, performance, or pipeline failures.
• Support testing, validation, and reconciliation of data during development, migration, and production phases.
• Follow security, governance, and compliance standards applicable to cloud data platforms.
• Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment.
• Maintain documentation for notebooks, workflows, data models, and migration approaches.


Responsibilities:
• Develop and maintain data engineering solutions using Azure Databricks and PySpark.
• Create, enhance, and optimize Databricks notebooks for data ingestion, transformation, and aggregation.
• Design and manage Delta Lake tables and pipelines supporting analytics and reporting use cases.
• Support cloud data migrations, including data validation and performance benchmarking.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with platform, DevOps, and data governance teams to ensure environment stability.
• Perform data pipeline testing and validation, ensuring correctness and completeness.
• Troubleshoot and resolve issues related to Spark jobs, Delta tables, and workflow execution.
• Participate in code reviews and enforce data engineering best practices.
• Support production deployments and post deployment stabilization.
• Provide inputs to data architecture and platform improvement initiatives.
• Mentor junior data engineers when required.

IF you are interested, please send me your updated resume ASAP with below details :


Full Name:


Current Location/Zip:


Contact Number:


E-Mail Id:


Alternate Email Id:


Visa/Work Permit Status:


Current Rate/Salary:


Expected Base Salary:


Notice Period/Availability to Start:


Skype ID:


Willingness to relocate to job location:


Any Relocation Concern (family/house/weather):


Current/Previous Employer Name:


Preferred Interview timings (Specify Time zone):


Any other job opportunity in process & at which stage:


Overall Experience Summary:


LinkedIn URL:


Looking forward for your response..

Thanks and Regards,


Abhishek Singh


Jconnect Infotech Inc.


168 Barclay Center Ste. 347,


Cherry Hill, NJ 08034


Contact: View phone number on click.appcast.io


Email: View email address on click.appcast.io