Azure DataBricks Developer

Azure Databricks Developer Location: Louisville, KY/ Chicago, IL/ Dallas, TX / Arlington, VA Fulltime Only Must Have Technical/Functional Skills

• Design, develop, and maintain cloud native data engineering solutions using Azure Databricks.
• Build and manage PySpark notebooks to process large scale structured and semi structured datasets.
• Design, create, and maintain Delta Lake tables, ensuring data reliability, ACID transactions, and schema enforcement.
• Develop scalable data workflows and pipelines using Databricks notebooks and orchestration patterns.
• Optimize performance of Spark jobs, including tuning partitions, memory usage, caching strategies, and query execution.
• Work extensively with PySpark and Spark SQL, choosing the appropriate approach based on use case and performance needs.
• Support cloud data migration initiatives, migrating data pipelines from on prem or legacy platforms to Azure Databricks.
• Integrate Databricks with upstream and downstream systems (e.g., data sources, storage layers, reporting tools).
• Ensure data pipelines are robust, reusable, and maintainable, following enterprise data engineering best practices.
• Implement error handling, logging, monitoring, and recovery strategies for production grade data pipelines.
• Collaborate with data architects, analysts, and downstream consumers to understand data requirements.
• Perform debugging and root cause analysis for data quality, performance, or pipeline failures.
• Support testing, validation, and reconciliation of data during development, migration, and production phases.
• Follow security, governance, and compliance standards applicable to cloud data platforms.
• Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment.
• Maintain documentation for notebooks, workflows, data models, and migration approaches.

Roles & Responsibilities

• Develop and maintain data engineering solutions using Azure Databricks and PySpark.
• Create, enhance, and optimize Databricks notebooks for data ingestion, transformation, and aggregation.
• Design and manage Delta Lake tables and pipelines supporting analytics and reporting use cases.
• Support cloud data migrations, including data validation and performance benchmarking.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with platform, DevOps, and data governance teams to ensure environment stability.
• Perform data pipeline testing and validation, ensuring correctness and completeness.
• Troubleshoot and resolve issues related to Spark jobs, Delta tables, and workflow execution.
• Participate in code reviews and enforce data engineering best practices.
• Support production deployments and post deployment stabilization.
• Provide inputs to data architecture and platform improvement initiatives.
• Mentor junior data engineers when required. AceStack LLC