Data Engineer

Data EngineerLocation: San Francisco CA Hybrid – Monday/Friday Remote + Onsite 595 Market St. #200, San Francisco, CA on Tuesday/Wed/Thursday Duration: Long TermRoles and ResponsibilitiesYou will build systems, core libraries and frameworks that power our batch and streaming Data and ML applications. The services you build will integrate directly with LendingClub's products, opening the door to new features.You will work with modern data technologies such as Hadoop, Spark, DBT, Dagster/Airflow, Atlan, Trino, etc., modern data platforms such as Databricks and Snowflake and cloud technologies across AWS stackBuild data pipelines that transform raw data into canonical schema representing business entities and publish it into the Data LakeImplement internal process improvements: automating manual processes, optimizing data delivery, reducing cloud costs, redesigning infrastructure for greater scalability, etc.Work with stakeholders including the Business, Product, Program and Engineering teams to deliver required data in time with high quality at reasonable costImplement processes and systems to monitor Data Quality, Observability, Governance and Lineage.Support operations to manage the production environment and help in resolving production issues with RCAWrite unit/integration tests, adopt Test-driven development, contribute to engineering wiki, and document design/implementation etc.Requirements4+ years of experience and a bachelor's degree in computer science or a related field; or equivalent work experienceWorking experience of distributed systems Hadoop, Spark, Hive, Kafka, DBT and Airflow/DagsterAt least 2 year of production coding experience in data pipeline implementation in PythonExperience working with public cloud platforms, preferably AWSExperience working with Databricks and/or SnowflakeExperience in Git, JIRA, Jenkins, shell scriptingFamiliarity with Agile methodology, test-driven development, source control management and test automationExperience supporting and working with cross-functional teams in a dynamic environmentYou have excellent collaborative problem solving and communication skills and are empathetic to othersYou believe in simple and elegant solutions and give paramount importance to qualityYou have a track record of building fast, reliable, and high-quality data pipelines