Data Bricks Migration and Support engineer

Data Bricks Migration and Support Engineer Seattle, WA Responsibilities Support post‑migration environment from IBM DataStage to Databricks. CI/CD Deployment: Support code deployments across Development, Test, and Production environments using Databricks Repos and REST APIs. Monitoring & Alerting: Set up monitoring via Databricks System Tables and observability tools to catch job failures, data anomalies, or latency spikes early. Workflow Management: Transition from DataStage job sequences to native Databricks workflows for scheduling, dependency tracking, and alerts. ETL Refactoring: Troubleshoot and fix issues in generated PySpark or Spark SQL code that replaced legacy DataStage Transformer or Lookup stages. Streaming & Batch Integration: Support ongoing data ingestion using Databricks Autoloader to process files continuously from cloud storage. Compute Management: Monitor and configure serverless or classic clusters to prevent over‑provisioning. Query Optimization: Analyze Spark execution plans, replace inefficient row‑by‑row logic with vectorized operations and native Spark functions. Storage Optimization: Maintain Delta Lake tables by enforcing layout optimization (ZORDER). Access Control: Implement granular permissions, column‑masking, and row‑level filters using Unity Catalog. Data Quality: Utilize Delta Live Tables to build pipelines with built‑in declarative data quality expectations and monitoring. Qualifications & Skills 8–15 years of experience in data migration or modernization to Databricks. Experience handling large migrations to Databricks, especially IBM DataStage to Databricks on AWS. Strong analytical skills to compare legacy and modern data platforms end‑to‑end. Understanding of Databricks Medallion‑layer architecture. Independently led and managed large Databricks migrations. Experience with CI/CD integration: version control (Git) and automated deployment processes for Databricks assets. Advanced SQL for building modular analytics workflows within Databricks SQL Analytics. Proficiency in Python or Scala for building, optimizing, and debugging complex data transformation scripts, ML pipelines. Experience in Apache Spark ecosystem: cluster execution flow, memory allocation, driver/worker nodes, data frames. Knowledge of Delta Lake architecture: ACID transactions, data skipping, partition strategies, automated data compaction. Experience with Delta Live Tables and Workflows for constructing production‑ready ETL pipelines. Experience with Unity Catalog for data governance, column‑/row‑level access control, data lineage. Experience with Auto Loader for incremental data ingestion. Skill in translating DataStage Parallel Jobs and Sequences into Python/PySpark scripts or Databricks notebooks. Ability to modernize legacy logic; adapt workflows to distributed DataFrames. Knowledge of mapping DataStage components to Spark operations. Experience building automated reconciliation frameworks for row counts, checksums, aggregates. Data cleansing experience: resolving type discrepancies, null handling, encoding issues. Orchestration: replace DataStage jobs with Databricks workflows or external orchestrators (Azure Data Factory, Airflow). Data governance expertise using Unity Catalog. Understanding of AWS cloud object storage, IAM, network security. Familiarity with Databricks Asset Bundles, CI/CD tools. Experience working in Agile teams and data governance frameworks. Excellent communication skills. Ability to collaborate with legacy and modernization application teams and stakeholders. Benefits Base Salary Range: $120,000 to $140,000 per annum. Discretionary Annual Incentive. Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans. Family Support: Maternal & Parental Leaves. Insurance Options: Auto & Home Insurance, Identity Theft Protection. Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Time Off, Sick Leave & Holidays. Legal & Financial Assistance: Legal Assistance, 401(k) Plan, Performance Bonus, College Fund, Student Loan Refinancing. #J-18808-Ljbffr Tata Consultancy Services