Associate Principal - Data Sciences

Job Title : Data ArchitectCertification Required Databricks Certified Associate Developer for Apache SparkJob SummaryWe are seeking a skilled PySpark Developer to design develop and optimize largescale data processing solutions using Apache Spark and Python The ideal candidate will have experience building ETL pipelines handling big data workloads and working with cloudbased data platforms to support analytics reporting and data engineering initiativesKey ResponsibilitiesDesign develop and maintain scalable data pipelines using PySparkBuild and optimize ETLELT processes for structured and unstructured dataProcess large datasets using Apache Spark in distributed environmentsCollaborate with business analysts data architects and data scientists to understand data requirementsIntegrate data from multiple sources including databases APIs and cloud storagePerform data cleansing transformation and validation activitiesOptimize Spark jobs for performance scalability and resource utilizationDevelop data quality checks and monitoring mechanismsTroubleshoot production issues and provide timely resolutionsImplement best practices for coding testing deployment and documentationWork with cloud platforms such as AWS Azure or GCP for data engineering solutionsParticipate in code reviews and ensure adherence to development standardsRequired SkillsStrong experience in Python and PySpark developmentHandson experience with Apache Spark Spark SQL DataFrames and Spark StreamingStrong knowledge of SQL and relational databasesExperience with ETL development and data warehousing conceptsFamiliarity with Hadoop ecosystem technologies HDFS Hive YarnExperience with version control systems such as GitUnderstanding of distributed computing concepts and performance tuningKnowledge of workflow orchestration tools such as Airflow Oozie or Azure Data FactoryStrong analytical and problemsolving skillsPreferred SkillsExperience with cloud platforms AWS EMR Azure Databricks Azure Synapse GCP DataprocExposure to Delta Lake Iceberg or Lakehouse architecturesExperience with Kafka or other realtime streaming solutionsKnowledge of CICD pipelines and DevOps practicesUnderstanding of data governance security and compliance requirementsQualificationsBachelors or Masters degree in Computer Science Information Technology Engineering or a related field58 years of experience in Data Engineering or Big Data developmentRelevant certifications in Spark Databricks Azure AWS or GCP are an added advantageNicetoHave TechnologiesDatabricksAzure Data Factory ADFAzure Synapse AnalyticsAWS GlueKafkaDelta LakeHiveAirflowSnowflakeSample Experience Requirement11 years of experience in Data Engineering11 years of handson experience with PySpark developmentExperience designing and supporting enterprisescale data platformsProven ability to optimize Spark applications for highvolume data processingKey CompetenciesCommunication and stakeholder managementProblemsolving and analytical thinkingTeam collaborationAttention to detailOwnership and accountabilityThis JD can be used for Data Engineer PySpark Senior PySpark Developer or PySpark Lead roles with minor modifications to experience and leadership expectations