Senior Data Engineer

Collaborate with data analysts, data scientists, ML engineers, software engineers, and business stakeholders to enable effective use of core data assets Design, develop, and maintain scalable data ingestion and transformation pipelines using Python, SQL, and modern data tooling Build and optimize data lake, lakehouse, warehouse, and data mart architectures Develop and maintain data models including facts, dimensions, feature datasets, and domain-specific data products Translate business requirements into design documents (e.g., ERDs, data flow diagrams) data models and ML feature pipelines Design and manage cloud-based data and ML infrastructure (Databricks preferred), including development, staging, and production environments Design, build, and operationalize machine learning pipelines for training, validation, deployment, and observability (e.g., performance, drift, reliability) Support ML model lifecycle management, including versioning, reproducibility, and lineage Develop and maintain ML feature stores and reusable feature pipelines for ML models Build and integrate AI-powered applications and agentic workflows (e.g., LLM-based agents, retrieval-augmented generation systems, workflow automation agents) Design and implement data pipelines for AI systems, including unstructured data (text, logs, embeddings, vector stores) Develop and maintain unit, integration, and data quality tests Participate in peer code reviews, pull requests, and team coding standards Document data pipelines, ML pipelines, models, infrastructure, and standard operating procedures Define infrastructure as code and support CI/CD pipelines for data and ML systems Ensure data privacy, security, and access control best practices (including AI data governance considerations) Identify and implement improvements in efficiency, scalability, resilience, and performance Contribute to evolving data, ML, and AI platform architecture, tools, and best practices Requirements Ability to gather requirements and translate business processes into data, ML, and AI solutions Comfortable working cross-functionally with both technical and non-technical stakeholders Ability to quickly learn new domains and technologies Strong Python development experience Advanced SQL development and query optimization skills Understanding of Databricks and large-scale data processing Experience building and scaling data pipelines using Databricks and PySpark Deep understanding of data lake, lakehouse, data warehouse, and data mart architectures Experience with data modeling across a variety of business domains Experience with modern data tooling (e.g., dbt or similar transformation frameworks) Knowledge of data formats, data patterns, and modeling best practices Experience with cloud platforms (AWS preferred) Experience with CI/CD pipelines in a data engineering environment Git-based development workflows Hands‑on experience with AI prompt and agent frameworks (e.g., Claude Code, Cursor, Windsurfer, or similar) Experience building AI agents and agentic workflows Exposure to LLMs, embeddings, vector databases, or generative AI systems Familiarity with handling structured and unstructured data (e.g., text, logs, embeddings) Experience building or supporting machine learning pipelines in production Familiarity with AI and MLOps in Databricks Experience with ML feature engineering and feature stores Understanding of ML model lifecycle management, monitoring, and evaluation Bachelor's degree in computer science, information systems, a quantitative field, or equivalent practical experience Core Competencies Demonstrates expertise in designing and developing scalable data ingestion and transformation pipelines using Python and SQL, while effectively collaborating with cross‑functional teams to translate business requirements into actionable data and ML solutions. Proficient in managing cloud‑based data infrastructure and operationalizing machine learning pipelines, ensuring data privacy and governance best practices. Highest-signal resume keywords Python Development SQL Development Databricks Experience Data Pipeline Development Machine Learning Lifecycle Management ATS Optimization Keywords Hard Skills Data Ingestion Data Transformation Data Modeling Feature Engineering CI/CD Pipelines Unit Testing Integration Testing Data Quality Testing AI Prompt Frameworks MLOps Soft Skills Cross‑Functional Collaboration Requirement Gathering Adaptability Certifications & Qualifications Bachelor's Degree in Computer Science Information Systems Quantitative Field Industry Keywords Data Lake Lakehouse Data Warehouse Data Mart AI Systems Unstructured Data Generative AI Data Governance ML Feature Stores Data Privacy Tools & Technologies Databricks AWS PySpark Dbt Git #J-18808-Ljbffr