Associate Principal - Data Engineering

Job Description Senior Developer – PySpark / Python Data Engineering Primary Skills: pysparkpython Developer Location: India, Global Delivery Center Regional Hub Industry: MultiNational FMCG Cloud Strategy: HyperscalerFirst Azure GCP AWS with Databricks Delta Lake Key Responsibilities PySpark Development Primary Focus Design and develop production‑grade PySpark applications for large‑scale batch and streaming data processing. Implement advanced PySpark DataFrame API operations: Complex transformations – Window functions, Pivot/Unpivot and nested struct handling. Multidataset joins – Broadcast joins, SortMerge joins and skew‑handling strategies. Custom UDFs – User‑Defined Functions, Pandas UDFs, Vectorized UDFs for performance‑critical transformations. Aggregations and GroupBy operations optimized for large FMCG datasets. Implement PySpark Structured Streaming for realtime data processing: Streaming sources – Kafka, Azure Event Hubs, GCP PubSub. Watermarking and windowing strategies for late‑arriving data. Stateful streaming operations using mapGroupsWithState. Exactly‑once and at‑least‑once delivery semantics. Apply advanced Spark performance tuning techniques: Partition optimization – repartition vs coalesce strategies. Handling data skew using salting and custom partitioners. Broadcast variable management and accumulator usage. Catalyst optimizer hints and AQE Adaptive Query Execution tuning. Executor sizing, memory fractions and parallelism configuration. Develop and maintain reusable PySpark libraries for shared data processing capabilities. Python Engineering Primary Focus Build Python‑based data services automation scripts and utility frameworks supporting the data platform. Develop REST API integrations using Python requests/ for consuming SAP OData, Salesforce and third‑party FMCG APIs. Implement data validation and reconciliation frameworks using Python Great Expectations, Pandera. Build Python‑based orchestration scripts and helper utilities for Airflow DAGs and Databricks Workflows. Apply software engineering best practices: Unit testing with pytest and integration testing with Testcontainers. Type hints, docstrings and modular design patterns. Virtual environments, dependency management (Poetry/pip) and packaging. Implement Python‑based data quality checks for completeness, consistency and conformity validations. Data Lakehouse Cloud Platform Primary Focus Build and manage Data Lakehouse architectures on hyperscaler platforms (Azure Databricks, GCP Dataproc, AWS EMR). Utilize Delta Lake, Apache Iceberg, Apache Hudi for ACID‑compliant data lake storage. Implement Medallion Architecture – Bronze/Silver/Gold for progressive data refinement. Use ACID transactions, schema enforcement, time travel, Optimize and ZOrder, Change Data Feed (CDF) for incremental data propagation. Manage Databricks Workflows and Job Clusters for production pipeline execution. Implement Databricks Auto Loader for incremental scalable data ingestion. Utilize Unity Catalog for data governance, lineage and access control. Data Ingestion Integration Build data ingestion pipelines from diverse FMCG data sources. Sources include SAP S/4HANA OData APIs, BAPI extracts, IDoc feeds; Salesforce REST API, Bulk API, Platform Events; Operational Databases – Oracle Cloud, SQL Azure, Cloud Spanner; Streaming sources – Apache Kafka, Azure Event Hubs, GCP PubSub; File‑based sources – SFTP, Azure Blob, GCS, S3, CSV, Parquet, Avro, JSON. Implement Change Data Capture (CDC) patterns for realtime database synchronization. Design schema evolution strategies to handle upstream data model changes gracefully. Publish processed data to downstream consumers – BigQuery, Azure Synapse, Snowflake, Power BI, Looker, Feature Stores (Feast, Databricks). SQL Data Modeling Write and optimize complex SQL queries for data extraction, transformation and validation. Design data warehouse schemas – Star and Snowflake models for FMCG analytics domains. Implement Spark SQL for large‑scale analytical query processing. Develop data quality SQL checks and reconciliation frameworks. Optimize SQL performance – Query plans, partition pruning, predicate pushdown. Benefits & Perks Comprehensive Medical Plan – covering medical, dental, vision. Short‑Term and Long‑Term Disability Coverage. 401(k) Plan with company match. Life Insurance. Paid Paternity and Maternity Leave. Vacation Time, Sick Leave, Paid Holidays. Salary and Compensation: The range displayed on each job posting reflects the minimum and maximum salary target for the position across all US locations. Within the range, individual pay is determined by work location and job level and additional factors including job‑related skills, experience, and relevant education or training. Depending on the position offered, other forms of compensation may be provided as part of overall compensation like an annual performance‑based bonus, sales incentive pay and other forms of bonus or variable compensation. Equal‑Opportunity Employer Statement LTIMindtree is an equal‑opportunity employer that is committed to diversity in the workplace. Our employment decisions are made without regard to race, color, creed, religion, sex (including pregnancy, childbirth or related medical conditions), gender identity or expression, national origin, ancestry, age, family‑care status, veteran status, marital status, civil union status, domestic partnership status, military service, handicap or disability or history of handicap or disability, genetic information, atypical hereditary cellular or blood trait, union affiliation, affectional or sexual orientation or preference, or any other characteristic protected by applicable federal, state, or local law, except where such considerations are bona fide occupational qualifications permitted by law. #J-18808-Ljbffr LTM