Databricks Platform Engineer

Data Engineering Role Data Engineering Responsibilities: Design and build batch data pipelines using Apache Spark on EKS and Databricks, processing data from NAS and upstream source systems into Iceberg tables on S3. Own the Bronze → Silver → Gold DLT pipeline architecture for assigned business domains. Write and maintain DDL for Iceberg tables following the CDP naming standard (catalog.schema.table with region, LOB, and layer components). Integrate with existing loading framework (Autosys → OpenShift → S3 → Airflow DAG trigger chain) and extend it as the platform evolves. Work with Data Stewards and DBA team to ensure schema design meets data classification requirements (INT, PII, CPII, CSPII). Implement and validate Unity Catalog governance controls — column masking, row-level filters, and lineage tracking — for tables under your ownership. Contribute to the migration of pipeline artifacts from DBFS to UC Volumes as part of the platform's governance uplift programmer DataOps. Responsibilities: Own the operational health of pipelines in your domain — monitoring, alerting, failure triage, and restart procedures. Build and maintain job monitoring integration between Databricks / Airflow and the team's observability stack (Splunk, CloudWatch, MSK audit events). Define and document incident runbooks for common failure patterns: EKS resource exhaustion, S3 VPC Endpoint connectivity, Spark executor out-of-memory, DLT pipeline schema evolution failures. Manage deployment of pipeline code and DAG updates through the Lightspeed CI/CD pipeline — understand what goes to S3 config bucket, what goes to the DAG bucket, and what goes to EFS. Participate in on-call rotation for production pipeline incidents. Work with the platform team on capacity planning for EKS cluster — there is no YARN queue here, resource exhaustion means job failure. Maintain data quality monitoring and reconciliation between on-prem and cloud outputs during the migration period. What We Are Looking For: Must Have: Overall 10 years' experience; 4+ years of data engineering experience, with at least 2 years working with Spark in a production environment. Solid understanding of cloud networking fundamentals — VPC, subnets, VPC Endpoints, Security Groups. You do not need to be a network engineer, but you need to understand why a Spark pod's network path matters and what breaks when it is wrong. Hands-on experience with AWS services: S3, EKS, IAM, KMS, SQS, MSK. Experience with Apache Iceberg or Delta Lake in production — not just awareness, but actual table design and maintenance. Familiarity with Kubernetes — you do not need to administer a cluster, but you need to understand pod lifecycle, resource limits, and how to read a pod describe output when something fails. Comfortable reading and writing Python and SQL at a production standard. Understanding of IAM and credential management — why static keys are unacceptable, how STS AssumeRole works, what Vault does. Strong Preference: Experience with Databricks, specifically Unity Catalog, MLflow, and DLT. Experience working in a regulated financial services environment — you understand why data classification, audit trails, and access controls are not optional extras. Familiarity with Airflow (DAG design, KubernetesPodOperator, trigger chain design). Experience with Spark performance tuning — you know what data skew looks like, when to add executors vs CPU, and how to read a Spark history server. Exposure to model governance or MLOps — understanding how MLflow Model Registry, Champion/Challenger alias management, and MRM review processes fit together. What You Will Not Be Doing: Building dashboards or BI reports. Managing the Databricks platform itself — that sits with the platform team. The Environment: Compute: EKS (Spark), OpenShift (on-prem loading framework), Databricks (UC, MLflow, DLT). Storage: S3 (Iceberg), EFS (intermediate artifacts), NAS (source files). Orchestration: Airflow (MWA), Autosys (on-prem, being phased out). Messaging: MSK (Kafka), SQS, SNS. Catalog: HMS on Aurora → migrating to AWS Glue → Unity Catalog. Security: Vault (credential injection), KMS (encryption), PrivateLink (no public internet for data paths), CloudTrail (audit). CI/CD: Lightspeed (Citi internal), Bitbucket / GitHub. What Good Looks Like in This Role: In the first 90 days, you should be able to: Explain the end-to-end data flow from NAS to an Iceberg table in S3, naming every component the data touches and the network path it takes. Run a Spark job on EKS independently, debug a failure from the pod logs, and know when to look at the Spark History Server vs CloudTrail. Identify a governance gap in a UC table configuration and propose a fix. In six months, you should own at least one business domain end-to-end — pipeline design, operational health, schema governance, and incident response. Salary Range: $125,000 to $140,000 per year Location: Irving, TX Function: TECHNOLOGY Role: Engineer

ID: 435866

Skills: Apache Spark Salary Range: $125,000 - $140,000 Desired Candidate Profile: BACHELOR OF COMPUTER SCIENCE Tata Consultancy Services