Senior Data Engineer - Azure Databricks / PySpark

Data Engineer / API Role With AI BuilderExperience Required: 4–6 yearsPrimary Role: Data Engineer / API Role With AI BuilderCloud Platform: Azure CloudRole Type: Individual ContributorCore Technical SkillsAzure DatabricksPySparkScalaAzure Data Factory (ADF)Azure Data Lake Storage Gen2 (ADLS Gen2)Delta LakeUnity CatalogPythonSQLGitHubCI/CDData Engineering ResponsibilitiesDesign, develop, and optimize scalable data pipelines.Build cloud-based data solutions using Azure.Develop ETL/ELT data processing workflows.Work with large-scale datasets in enterprise environments.Perform data modeling and performance tuning.Implement and maintain Delta Lake solutions.Work with Databricks, ADF, and PySpark for data engineering workloads.Develop end-to-end data solutions independently.Apply data governance and security best practices.API Development SkillsStrong hands-on experience building REST or GraphQL APIs.Understanding of API design and contracts, API security, error handling, and API performance optimization.Experience deploying APIs using Kubernetes, preferably Azure Kubernetes Service (AKS).AI / ML SkillsExposure to AI/LLM technologies.Experience or exposure to agentic frameworks.Familiarity with Databricks AI/ML, Codex, Claude, and other LLM/AI development frameworks.Cloud & DevOpsStrong hands-on experience with Azure Cloud.Experience with CI/CD practices.GitHub experience.Knowledge of cloud-native data platforms.Understanding of Kubernetes/AKS deployment.Data Governance & SecurityStrong understanding of data governance.Knowledge of data security and access control.Experience with Unity Catalog is preferred.Ability to implement secure and governed data solutions.Essential SkillsDatabricksAzure Data Factory (ADF)PySparkAzure CloudADLS Gen2Delta LakeDesirable SkillsScalaREST APIs / GraphQLKubernetes / AKSAI/LLMAgentic frameworksDatabricks AI/MLCI/CDUnity CatalogKey ResponsibilitiesOwn end-to-end solution delivery.Analyze and solve complex data engineering problems.Collaborate with cross-functional teams and business stakeholders.Build scalable and reliable data pipelines.Optimize data processing and API performance.Follow enterprise standards for governance, security, and deployment.