Senior Python/Pyspark Data Engineer

Greetings!We are seeking a senior Python / PySpark Data Engineer to join a high-impact data engineering initiative for a leading global financial institution in Whippany, NJ. In this role, you will design, engineer, and optimize distributed data pipelines and analytics engines supporting risk calculations, regulatory compliance reporting, transaction processing, and enterprise data lake modernization.Job Title: Python / PySpark Data EngineerDomain: Banking & Financial ServicesLocation: Whippany, NJ (Hybrid: 3 Days Onsite, 2 Days Remote)Work Authorization: Open to all authorized candidates (W2 only)Interview Process: In-Person Interview onsite at the Whippany, NJ campus.Key ResponsibilitiesDesign, develop, and maintain high-throughput batch and real-time distributed data pipelines using Python and PySpark.Ingest, transform, and clean multi-terabyte financial datasets across structured, semi-structured, and unstructured sources.Optimize PySpark jobs by analyzing DAG execution plans, tuning memory management, addressing data skew, and refining partition strategies.Implement robust data quality checks, reconciliation routines, and audit logging to ensure strict adherence to banking data governance standards.Collaborate with upstream transaction systems and downstream consumption layers (BI dashboards, risk analytics models, and regulatory platforms).Build automated CI/CD deployment pipelines, automated testing frameworks, and monitoring alerts for production data jobs.Partner with engineering lead in Agile standups, code reviews, and architectural design sessions during onsite collaboration days.Required Technical Skills & QualificationsProfessional Experience6 to 9+ years of overall software and data engineering experience, with a solid footprint in Banking, Capital Markets, or Financial Services.Python & PySparkAdvanced proficiency in Python (OOP, collections, multiprocessing) and hands-on expertise building scalable pipelines with PySpark (DataFrames, Datasets, Spark SQL).Performance TuningProven ability to debug and optimize Spark applications (broadcast joins, caching, shuffle partition tuning, Catalyst optimizer analysis).Database & QueryingStrong SQL writing skills with experience working with relational databases (PostgreSQL, Oracle) as well as modern analytical data warehouses (Snowflake, Databricks Delta Lake, or Redshift).Cloud & OrchestrationPractical familiarity running workloads in public cloud environments (AWS/Azure) utilizing services like S3, EMR/Glue, and workflow orchestrators (Apache Airflow, Step Functions).Financial Domain KnowledgeUnderstanding of banking data domains such as Trade Lifecycle, PnL, Market/Credit Risk, Fixed Income, or Regulatory Reporting (e.g., CCAR, Basel).