Hadoop,Pyspark,Hive,Kafka - Senior Developer

Job Description Seeking a Senior Big Data Engineer with 10–13 years of experience specializing in Hadoop, PySpark, Kafka, Hive, and strong experience designing data solutions for large-scale financial systems. This role focuses on delivering highly performant, well-governed data platforms that support the bank’s mission‑critical global markets functions. Delta Change Data Feed (CDF): schema evolution & enforcement, ACID transaction handling Roles & Responsibilities Primary responsibilities include designing, developing, and optimizing PySpark-based ETL pipelines on on‑prem Hadoop clusters and cloud environments; building real‑time streaming ingestion frameworks with Kafka; managing Hadoop ecosystem components (HDFS, YARN, MapReduce, Tez, Oozie, Airflow); and creating Hive data models for regulatory reporting and market risk. Key Responsibilities Design, develop, and optimize PySpark-based ETL pipelines. Build high‑volume ingestion frameworks using Kafka for real‑time and near‑real‑time trading and market data. Develop, tune, and manage Hadoop ecosystem components—HDFS, YARN, MapReduce, Tez, Oozie, Airflow. Build high‑performance, optimized Hive data models for regulatory reporting, trade lifecycle, and market risk processing. Databricks Lakehouse & Delta Framework Architect and implement Bronze/Silver/Gold layer modeling patterns within the Databricks Lakehouse. Apply Delta Lake best practices: optimized file management Z‑Ordering Delta Change Data Feed: schema evolution & enforcement, ACID transaction handling Build reusable frameworks for ingestion, cleansing, transformation, and consumption of data across Lakehouse layers. Enable governance, lineage, and auditability using Unity Catalog or equivalent cataloging tools. Collaboration, Leadership & Delivery Collaborate closely with quants, product owners, architects, risk tech, and business users. Participate in agile ceremonies—sprint planning, refinement, design reviews. Mentor junior engineers and contribute to building strong engineering practices across tech teams. Required Skills & Experience 10–13 years of hands‑on experience in Big Data engineering. Expert skills in: PySpark—dataframe optimizations, partitioning, broadcast strategies, distributed computing. Kafka—producer/consumer design, schema registry, streaming ETLs. Hadoop ecosystem—HDFS, YARN, MapReduce/Tez, Oozie, Airflow. Hive—advanced query tuning, TEZ optimization, partition/bucket management. Databricks Lakehouse—Bronze/Silver/Gold layer modeling, Delta Lake optimizations, data quality frameworks, structured & unstructured data handling. Experience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting. Strong SQL knowledge for massive datasets (TB/PB scale). Experience with CI/CD practices—Git, Jenkins, Bitbucket, build pipelines. TCS Employee Benefits Summary Discretionary Annual Incentive. Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans. Family Support: Maternal & Parental Leaves. Insurance Options: Auto & Home Insurance, Identity Theft Protection. Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Sick Leave & Holidays. Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing. Salary Range: $110,000–125,000 a year. Qualifications

BACHLOR OF COMPUTER SCIENCE

#J-18808-Ljbffr Tata Consultancy Services