Azure Databricks Engineer

Data Engineer

Key Responsibilities:

• Design and implement scalable data processes focused on fulfilling product requirements.
• Large complex components; influencing overall product architecture and patterns.
• Autonomously implement component design in line with pre-defined Data and (ELT/ETL) architectural patterns.
• Partner with Business, Technical and Strategic Product stakeholders to manage project commitments in an agile framework; rapidly delivering value to our customers via technology solutions.
• Develop, construct, test, document and maintain data pipelines.
• Identify ways to improve data reliability, efficiency, and quality.
• Design and develop resilient, reliable, scalable and self-healing solutions to meet and exceed customer requirements.
• Ensure that all parts of the application eco-system are thoroughly and effectively covered with telemetry.
• Focus on automation, quality and streamlining new and existing data processing.
• Create data monitoring capabilities for each business process and work with data consumers on updates to data processes.
• Develop data pipelines using Python, Spark and/or Scala.
• Automate and orchestrate data pipelines using Azure Data Factory or Delta Live Tables
• Help maintain the integrity and security of the company data.
• Communicate clearly and effectively in oral and written forms and be able to present and demonstrate work to technical and non-technical stakeholders.

Required Qualifications:

• Undergraduate degree or equivalent experience
• Minimum 5 to 7 years of IT experience in Software Engineering.
• Minimum 3 to 5 years of experience in big data processing for batch and/or streaming data; data includes file systems, data structures/databases, automation, security, messaging, movement, etc.
• Minimum 3 to 5 years of experience with Python and Spark in developing data processing pipelines.
• Minimum 3 to 5 years of ETL programming experience in Databricks using Scala/Java.
• Minimum 3 to 5 years of supporting extensive data analysis using advanced SQL concepts and window functions
• Minimum 2 to 3 years of experience working on large scale programs, with multiple concurrent projects.
• Minimum 2 to 3 years of experience with Agile methodologies and Test-Driven Development.
• Experience with Continuous Development/ Integration (CI/CD) /DevOps skills, (Jenkins, Git, Azure DevOps/ Git Actions).
• Strong written and oral communications along with presentation and interpersonal skills
• Ability to lead and delegate work across other members of the data engineering team

Preferred Qualifications:

• Minimum 2 to 3 years of cloud experience, preferably Azure.
• Minimum 2 to 3 years experience with Databricks or other big data platforms.
• Minimum 2 to 3 years automation/orchestration experience using Azure Data Factory
• Experience and familiarity with data across Healthcare Provider domains (i.e. Patient, Provider, Encounter, Billing, Claims, Eligibility etc.)
• Familiarity/Experience with Clinical data exchange formats like HL7, FHIR, CCD etc.
• Familiarity/experience with FHIR resources and data model
• Experience in Big data processing in healthcare domain
• Cloud development and computing.
• Knowledge of cutting to edge technologies (AI, ML, Blockchain, Wearables, IOT).
• Knowledge/Experience with Microservice design, Java Spring, Kafka, RabbitMQ.