Data Engineering Lead

Job Description At Truckstop, we have transformed the entire freight-moving lifecycle with our SaaS solutions. From freight matching to payments and everything in between, we are the trusted partner for carriers, brokers, and shippers alike. We lead this industry forward with our One Team mindset committing to principles such as assume positive intent, have each other's back, and be your authentic self. Our drive for greatness produces high expectations, yet our regard for humans is even higher. Join a team of brilliant minds and generous hearts who care deeply about other's success. Position Summary: The Data Engineering team within Data Services is responsible for creating and maintaining data pipelines, assembling complex datasets, and extracting, transforming, and loading data from a variety of data sources. Data Engineers work closely with Software Engineers, Product Managers, Data Scientists, and teams across the enterprise to deliver product features, build data pipelines, and architect our Product and Analytics data infrastructure for optimal performance. Data Engineers ensure that Truckstop has hygienic, accurate, data available at the point of need and partner with the System Architecture and DevOps teams to troubleshoot data pipeline production issues and respond to incidents. The SR/Lead Data Engineer role works directly with the Director of Data Services to oversee the Data Engineering team, handle the most complex issues, and provide coaching and guidance to more junior Data Engineers. The Sr/Lead DE possesses expert working knowledge of technical subject matter, as well as the ability to strategize and problem solve on data infrastructure, data migration tooling, data modeling, and complex troubleshooting. Essential Job Functions: Create, automate, and maintain complex data pipelines from multiple source systems using migration and orchestration technology, employing architecture best practices, leveraging leading edge technology, and utilizing multiple programming languages. Extract, transform, and load data from a variety of data sources using SQL and Python and cloud data technologies. Interact with API and Kafka technology to ingest and transform data into Snowflake for Product and Data Science team end uses. Write and optimize queries and data processes, write orchestration functions, perform source to target mapping and data modeling. Work directly with Product to engineer performant data pipelines required to support Product analytics, reporting, and strategy. Work with data architects, database administrators, Infosec and software engineering solution architects to continually improve data security, data capture, data pipelines, and data infrastructure. Work with IT operations to resolve data related technical issues and respond to data related major incidents. Perform incident resolution and root cause analysis of critical outages. Implement solutions to systematic failures. Provide on-call support, including after-hours. Assist with documentation of the environments that support our products in Confluence and Atlan Position Requirements: Bachelor's degree or equivalent professional experience required; Computer Science or Engineering preferred. Demonstrated expertise in SQL and Python required. Minimum of +6 years of experience required. Experience with Snowflake, MS SQL, and PostgreSQL required. Experience building and optimizing reliable, idempotent data pipelines (streaming or batch) using ETL tools such as Matillion, Apache Airflow, FiveTran, Kafka and Spark required. A successful history applying software engineering principles to build data infrastructure tools/libraries to automate and scale data pipelines. Experience with supporting Data Science applications, including machine learning data pipelines; Azure ML experience a plus Understanding of applying logging and metrics to monitor and detect data pipeline performance issues. Advanced understanding of data warehousing architecture including performance optimization and tuning, specifically within Snowflake and Azure. Working knowledge of dbt data modeling and source to target mapping with a demonstrated understanding of data schema design, Snowflake clustering and micro-partitioning Experience working in Azure and AWS cloud-based environment At Truckstop we are dedicated to creating a workplace that is equitable, inclusive, and inspiring for every employee. In an effort to provide greater transparency, we are sharing the base salary range for this position. The position is also eligible for a yearly bonus. Final salary is based on a number of factors including market location, job-related knowledge, education/training, certifications, key skills, experience, internal peer equity as well as business considerations. The anticipated base pay range for this position is $140,000—$160,000 USD. The above description covers the most significant duties performed but does not include other related occasional work that may be assigned or is completed by the employee. Truckstop provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state, or local laws. Truckstop participates in the E-Verify program. Learn more about the E-Verify program here: #J-18808-Ljbffr