Data Engineer

Data EngineerSan Francisco, CA - There is a preference for candidates to be in San Francisco, but remote options are ok.Long termRequired SkillsSQL, Python, Spark or PySpark, and Scala. Databricks. ETL Tools, preferably Datastage, but others like FiveTran or Informatica are ok.BI Reporting Tools - SASS Cubes, Tableau, ThoughtSpot, Power BI, etc. Platform Migration experience.Strong communication and cross-functional support skills.Additional SkillsTeam is in charge of all Data Engineering related to BI work for the whole company. Currently they have 2 platforms, an older SQL Warehouse and a newer Databricks-based platform.Beyond the day to day of supporting the whole organizations BI based Data Engineering needs, they will be working on migrating entirely to the new platform.This resource needs to be Senior and able to handle multiple tasks.Job DescriptionContinuously improve our data infrastructure and stay ahead of technology.Build strong cross-functional partnerships with Data Scientists, Analysts, Product Managers and Software Engineers to understand data needs and deliver on those needs.Evaluate and select appropriate BI tools and technologies based on the organization's needs. Run POC and the implementation and configuration of BI software, including data visualization tools, reporting platforms, and analytics solutionsStay updated with the latest trends, technologies, and best practices in the BI field. They explore opportunities for process improvement, automation, and innovation to enhance the organization's data analytics capabilitiesEstablish Data governance policies and procedures to ensure data quality, consistency, and security. Define access controls, data privacy measures, and data retention policies to comply with regulatory requirements and protect sensitive informationWe’re excited about you if you have:8 -10 years of experience in data engineering and designing scalable ETL pipelinesAdvanced Proficiency in Hadoop, Scala, Spark, Kafka, and SQL6+ years of experience with cloud platforms like Azure, AWS, GoogleBe skilled at architecting large and complex data pipelines.Experience performing data analysis and data exploration.Experience working with Cloud technologies, Azure, Databricks, ADLS, Spark, Cosmos DB, and other big data technologies.Experience working in an agile delivery environment.Experience working in multi-developer environments, using version control.Excellent knowledge of data structures and design patterns?Preferred experience with data integration toolsWork with cross-functional agile teams to drive projects through full development cycle.Help the team improve with the usage of data engineering best practices.Write SQL for processing raw data, data validation, and QAKnowledge working with APIs to collect or ingest dataStrong Database knowledge, SQL & No-SQL preferredCommunication Skills Data Engineers are part of a team, working with database administrators, data analysts and management, and need to be effective communicators.Problem-Solving Skills Data Engineers look at an issue that needs to be solved and come up with solutions quickly.Experience integrating and building data platforms in support of BI, Analytics, Data Science, and real-time applications.Strong communication skills, with the ability to initiate and drive projects proactively and accuratelyExperience with Datastage or other ETL tools, experience with reporting tools like Cognos, SSAS Cubes, Tableau, Thoughtspot.