Senior DataOps Engineer

Senior DataOps EngineerJoin a cutting-edge enterprise data platform built on Databricks and Azure, where you'll play a pivotal role in developing and maintaining a self-healing, AI-enabled DataOps environment. This fully remote position offers the opportunity to work at the forefront of data infrastructure innovation, leveraging AI-driven observability and automation to optimize platform reliability and scalability.Key ResponsibilitiesDesign and implement AI-powered observability workflows to proactively detect pipeline drift, anomalies, and SLA risks.Integrate AI tools such as Databricks Mosaic AI, Genie, and OpenAI APIs into operational DataOps processes to enhance automation and diagnostics.Develop AI-assisted root cause analysis tools to reduce MTTR and improve platform learnings.Manage the full deployment lifecycle of data pipelines and platform configurations across development, staging, and production environments using GitHub Enterprise and Linear.Operate and optimize the full Azure data services stack, including ADLS Gen2, ADF, Azure Monitor, and Key Vault.Architect and govern data pipelines on Databricks using Delta Live Tables, PySpark, and Workflows, ensuring adherence to platform standards.Tune Spark jobs for performance, reliability, and cost efficiency, managing cluster configurations and resource utilization.Enforce data quality and governance standards across all data layers, collaborating with data governance teams on data contracts and SLAs.Produce comprehensive technical documentation, including runbooks, deployment procedures, and incident post-mortems.Partner with cross-functional teams to align platform and pipeline design with business needs and technical best practices.Required QualificationsExtensive hands-on experience with Databricks, including Delta Live Tables, Unity Catalog, Spark performance tuning, and Delta Lake internals.Deep expertise in Azure data services: ADF, ADLS Gen2, Azure Monitor, Log Analytics, Key Vault.Proven experience integrating AI tools into DataOps workflows, such as LLMs, anomaly detection, and intelligent diagnostics.Strong CI/CD experience using GitHub Enterprise, including environment promotion, branch strategies, and release management.Proficiency in Python and/or Scala for pipeline development, with SQL fluency for data transformation and validation.Hands-on experience designing and orchestrating large-scale ADF pipelines, especially in multi-tenant environments.Demonstrated ability to operate independently, solve complex problems, and communicate effectively with technical and business stakeholders.Preferred QualificationsExperience with Linear for sprint planning and release tracking.Familiarity with Databricks Mosaic AI, Genie, or other AI-native Databricks features.Exposure to agentic AI frameworks or MCP server integrations.Background working with real estate, property management, or multi-source ERP data environments.Knowledge of data governance frameworks, data lineage, and metadata management within Unity Catalog.About Delphi-USPeacemakers in the Talent War Delphi-US is a national IT Services firm based in Newport, Rhode Island. Delphi attracts our nation's best and brightest Technical & Professional Talent and positions our nation's finest employers to further their competitive advantages. Delphi's core technical focus includes Data Intelligence Engineering & Analytics, AI/ML Operational Excellence, DevOps & Cloud Computing, Health IT & Life Sciences, and Energy Management/Power Systems; serving domains from sensitive national energy sectors to critical financial services, biomed, life sciences & pharma, major government & municipal institutions, and much of the USA's Fortune 500 corporations. Delphi Associates are deployed on critical IT Projects where their consultative business acumen, comprehensive technical expertise, and committed service mission drives our client's success. Our proprietary skill-based and cultural matching process results in higher qualified project teams, custom curated for our client's specific technical & functional goals. We are supremely experienced, friendly, professional, and ready to advocate on your behalf. Delphi has an indelible understanding of employer expectations, a track record of project success and we deliver lasting results. 6395