Associate Principal - Data Engineering

Job Title : Senior Big data DeveloperJob Location : Irving,TexasJob SummarySeeking a Senior professional with 10 years of experience in Python and Spark SQL to design and optimize scalable big data processing solutions within the DatabricksSparkSQLPython ecosystemJob DescriptionDevelop and maintain scalable data processing pipelines using Apache Spark with Python and Scala Utilize SparkSQL to analyze large datasets and extract meaningful insights supporting datadriven decision making Collaborate with crossfunctional teams to integrate big data solutions into existing systems Optimize Spark jobs for performance and efficiency in distributed computing environments Ensure data quality and consistency across various data sources Stay updated with the latest advancements in Spark Scala Python and related big data technologies to continuously improve processing capabilitiesRoles and ResponsibilitiesDesign develop and deploy complex data processing workflows leveraging Apache Spark Scala Python and SparkSQL Write efficient reusable and maintainable code for Spark applications Troubleshoot and resolve performance bottlenecks in Spark jobs and data pipelines Collaborate with data engineers data scientists and stakeholders to translate business requirements into technical solutions Conduct code reviews and provide mentorship to junior team members on best practices and coding standards Participate in architectural discussions and contribute to the evolution of big data platforms Monitor and maintain the health availability and reliability of Spark clusters and data servicesKarat Interview mandate