Azure DataBricks Engineer

Role - Azure DataBricks Engineer
Location - Louisville, KY/ Chicago, IL/ Dallas, TX / Arlington, VA
Duration - Fulltime Role

Must Have Technical/Functional Skills
• Design, develop, and maintain cloud native data engineering solutions using Azure Databricks.
• Build and manage PySpark notebooks to process large scale structured and semi structured datasets.
• Design, create, and maintain Delta Lake tables, ensuring data reliability, ACID transactions, and schema enforcement.
• Develop scalable data workflows and pipelines using Databricks notebooks and orchestration patterns.
• Optimize performance of Spark jobs, including tuning partitions, memory usage, caching strategies, and query execution.
• Work extensively with PySpark and Spark SQL, choosing the appropriate approach based on use case and performance needs.
• Support cloud data migration initiatives, migrating data pipelines from on prem or legacy platforms to Azure Databricks.
• Integrate Databricks with upstream and downstream systems (e.g., data sources, storage layers, reporting tools).
• Ensure data pipelines are robust, reusable, and maintainable, following enterprise data engineering best practices.
• Implement error handling, logging, monitoring, and recovery strategies for production grade data pipelines.
• Collaborate with data architects, analysts, and downstream consumers to understand data requirements.
• Perform debugging and root cause analysis for data quality, performance, or pipeline failures.
• Support testing, validation, and reconciliation of data during development, migration, and production phases.
• Follow security, governance, and compliance standards applicable to cloud data platforms.
• Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment.
• Maintain documentation for notebooks, workflows, data models, and migration approaches.


Roles & Responsibilities
• Develop and maintain data engineering solutions using Azure Databricks and PySpark.
• Create, enhance, and optimize Databricks notebooks for data ingestion, transformation, and aggregation.
• Design and manage Delta Lake tables and pipelines supporting analytics and reporting use cases.
• Support cloud data migrations, including data validation and performance benchmarking.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with platform, DevOps, and data governance teams to ensure environment stability.
• Perform data pipeline testing and validation, ensuring correctness and completeness.
• Troubleshoot and resolve issues related to Spark jobs, Delta tables, and workflow execution.
• Participate in code reviews and enforce data engineering best practices.
• Support production deployments and post deployment stabilization.
• Provide inputs to data architecture and platform improvement initiatives.
• Mentor junior data engineers when required.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.