AWS Databricks Lead

AWS services and databrick Location Hartford CT onsite Proficient in leveraging Databricks features such asoDelta LakeoUnity CatalogoScalable data processing frameworksStrong understanding of AWS cloud services includingoAmazon S3oAWS LambdaoAmazon SNSoAWS Step FunctionsoAWS GlueSkilled in job cost optimization techniques tooMinimize runtimesoImprove system performanceExperience withoPython core librariesoPySpark for data engineering tasksGood analytical and problemsolving skills with expertise inoSQL query developmentoStored proceduresoCreating data marts and views in RDBMSStrong understanding ofoETL conceptsoEndtoend data pipeline architectureoData warehouse and reporting layersFamiliarity with CICD pipelines and deployment tools such asoTerraformExperience with version control systemsoGitHub or equivalent platformsUnderstanding ofoAgile methodologyoSoftware Development Life Cycle SDLCExcellent communication and presentation skills with the ability to collaborate effectively with stakeholdersRoles ResponsibilitiesCollaborate with business users and stakeholders to understand challenges and provide effective Databricksbased solutionsResearch and evaluate new Databricks features conduct Proof of Concepts POCs and develop best practice documentationProvide support for Sparkrelated issues and participate in code reviews offering recommendations for improvementLead and mentor junior team members guiding them in development and troubleshooting activitiesManage midsized teams and oversee delivery of work ensuring quality and timely completionTake ownership of assigned tasks and contribute effectively in an Agile environmentParticipate in solution design discussions and prepare designdocumentation artifacts based on requirementsDrive job cost optimization and performance tuning initiatives to enhance system efficiencySupport CICD processes and infrastructure automation to streamline deploymentsActively participate in Agile ceremonies and contribute to continuous improvement of team collaboration and productivity