Data Engineering Lead

Actively collaborate with Principal Software Engineers and Data Architects to define solution architecture Lead the solution design and optimization of scalable data pipelines and microservices in Python, enabling both real-time and batch data processing across enterprise platforms Drive the development of cloud-native data ingestion and streaming solutions leveraging AWS services including Kinesis, S3, DynamoDB, EventBridge, and related technologies Own the design, implementation, and operational excellence of data integration frameworks and source data pipelines supporting CI Autonomy initiatives Partner with business, product, and engineering stakeholders to translate complex requirements into scalable data architectures, workflows, mappings, and system designs Establish and enforce automated testing, data quality controls, and validation frameworks to ensure integrity, reliability, and compliance across distributed data ecosystems Lead operational monitoring, performance tuning, and root-cause analysis of production data platforms using observability tools such as CloudWatch to maintain high availability and service reliability Bachelor’s degree in Computer Science, Computer Engineering, or related field 8+ years of experience in data engineering or related disciplines with increasing responsibility Extensive experience on modern, large scale, complex Caterpillar data platforms such as Helios Data Platform Strong foundation developing and deploying Python solutions to a production environment Experience leading teams to build high-throughput, scalable data pipelines Strong hands-on experience with AWS data services (Kinesis, S3, DynamoDB, EventBridge, etc.) at scale Strong in SQL, including data quality and validation practices Experience in deploying software using CI/CD tools such as Azure DevOps, Jira, Jenkins, etc. Experience developing microservices that support real-time data ingestion Experience developing software applications using relational and noSQL databases Ability to ensure data integrity across distributed and streaming systems Experience with monitoring, testing, and automation in large-scale data environments The pay range that the employer in good faith reasonably expects to pay for this position is $52.01/hour - $81.26/hour. Our benefits include medical, dental, vision and retirement benefits. Applications will be accepted on an ongoing basis. We are an equal opportunity employer, and we do not discriminate on the basis of race, religion, color, national origin, sex, sexual orientation, age, veteran status, disability, genetic information, or other applicable legally protected characteristic. Qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable law, including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Unincorporated LA County workers: we reasonably believe that criminal history may have a direct, adverse and negative relationship with the following job duties, potentially resulting in the withdrawal of a conditional offer of employment: client provided property, including hardware (both of which may include data) entrusted to you from theft, loss or damage; return all portable client computer hardware in your possession (including the data contained therein) upon completion of the assignment, and; maintain the confidentiality of client proprietary, confidential, or non-public information. In addition, job duties require access to secure and protected client information technology systems and related data security obligations. #J-18808-Ljbffr