Senior AI Data Pipeline Engineer

We Are Looking For The Best

42dot AI . (PB) GPU, AI.

At 42dot, our AI Data Pipeline Engineer architect and scale global data pipelines that ingest and process data from worldwide sources. You will design and operate high-throughput systems to reliably deliver petabyte-scale data to our large-scale GPU infrastructure, powering mission-critical AI workloads.

Responsibilities

• Design and build high-performance, scalable data pipelines to support diverse AI and Machine Learning initiatives across the organization.
• Architect and implement multi-region data infrastructure to ensure global data availability and seamless synchronization.
• Develop flexible pipeline architectures that allow for complex branching and logic isolation to support multiple concurrent AI projects.
• Optimize large-scale data processing workloads using Databricks and Spark to maximize throughput and minimize processing costs.
• Maintain and evolve the containerized data environment on Kubernetes, ensuring robust and reliable execution of data workloads.
• Collaborate with AI researchers and platform teams to streamline the flow of high-quality data into training and evaluation pipelines.

Qualifications

• AI/ML
• Apache Spark Databricks
• Apache Airflow
• Kubernetes
• Apache Kafka
• Python
• Best practices
• Extensive professional experience in building and operating production-grade data pipelines for massive-scale AI/ML datasets.
• Strong proficiency in distributed processing frameworks, particularly Apache Spark and the Databricks ecosystem.
• Deep hands-on experience with workflow orchestration tools like Apache Airflow for managing complex dependency graphs.
• Solid understanding of Kubernetes and containerization for deploying and scaling data processing components.
• Proficiency in distributed messaging systems such as Apache Kafka for high-throughput data ingestion and event-driven architectures.
• Expert-level programming skills in Python for system-level optimizations.
• Strong knowledge of cloud-native services and best practices for building secure and scalable data infrastructure.
• Logical approach to problem-solving with the persistence to identify and resolve root causes in complex, large-scale systems.
• Strong communication skills to effectively collaborate with cross-functional teams and external partners.

Preferred Qualifications

• / (Latency)
• Ray AI
• Spark Streaming Flink /(Near real-time)
• Terraform Infrastructure as Code(IaC)
• ML (MLOps)
• Experience in architecting global, multi-region data pipelines and solving challenges related to cross-border data transfer and latency.
• Practical experience or a strong interest in implementing distributed computing frameworks like Ray for AI workloads.
• Experience in building real-time or near-real-time pipelines using Spark Streaming or Flink.
• Familiarity with Infrastructure as Code (IaC) tools such as Terraform to manage complex data environments.
• Understanding of the end-to-end ML lifecycle (MLOps) and how data infrastructure supports model experimentation and deployment.

Interview Process

• Resume Screening - Coding Test - Virtual Interview (approximately 1 hour) - Onsite or Virtual Interview (approximately 3 hours) - Final Offer
• Please note that the interview process may vary depending on the position and is subject to change based on scheduling and other circumstances.
• Interview schedules and results will be communicated individually via the email address provided in your application.

Additional Information

• Please upload all required documents in PDF format.
• Veterans and applicants eligible for employment protection will receive preferential consideration in accordance with applicable laws and regulations.
• In compliance with the Act on Employment Promotion and Vocational Rehabilitation for Persons with Disabilities, registered individuals with disabilities will receive preferential consideration.
• 42dot does not accept unsolicited resumes from search firms. We will not pay any fees for resumes submitted without prior agreement.
• A 3-month probationary period may apply.