Data / AI Native Engineer

Role: Data / AI Native Engineer

Location: NYC, NY/ NJ (Hybrid)

Type: Contract

Day-to-Day Job Duties:

• Design, build, optimize, and support enterprise-scale Databricks and AWS Lakehouse data pipelines using Spark and PySpark.
• Develop Databricks solutions leveraging Delta Lake, Medallion Architecture (Bronze/Silver/Gold), and Unity Catalog.
• Develop high-volume batch and streaming pipelines using PySpark, ensuring scalability, performance, reliability, data quality, and maintainability.
• Design and implement modern lakehouse engineering patterns including SCD Type 2, Change Data Capture (CDC), idempotent writes, schema evolution, and data contracts.
• Work across the AWS data ecosystem, including technologies such as S3, Glue, Lambda, EMR, Redshift, Kinesis, and related AWS services.
• Use GitHub Copilot and/or Claude Code CLI as part of the daily engineering workflow to accelerate application code, data-pipeline development, testing, debugging, optimization, and documentation.
• Rapidly develop proofs of concept and MVPs using AI-assisted development, while establishing clear paths to scalable production implementation.
• Apply AI capabilities directly within the data-engineering lifecycle, including use cases such as:
• AI-assisted schema inference
• Automated data-quality rule generation
• LLM-based anomaly detection
• AI-generated PySpark transformations
• Intelligent data validation and enrichment
• Build data foundations for AI-enabled products, including feature stores, vector stores, RAG data layers, embedding pipelines, and other AI-ready data architectures.
• Design and develop high-volume streaming pipelines using technologies such as Spark Structured Streaming, Kafka, Kinesis, and Databricks Auto Loader.
• Develop and orchestrate enterprise data pipelines using technologies such as Databricks Workflows, Delta Live Tables, Airflow, and dbt.
• Implement secure and governed data-engineering practices using Unity Catalog, including catalog/schema organization, access control, lineage, governance, and data discovery.
• Lead technically complex data-engineering initiatives, drive solution design, coordinate engineering activities, and help resolve complex platform and pipeline issues.
• Partner with architects, software engineers, product owners, source-system teams, governance teams, analytics teams, AI teams, and business stakeholders.
• Remain actively hands-on with PySpark, Databricks, AWS, and software-engineering activities, rather than operating solely as a people manager or high-level data architect.

Basic Qualifications:

• Minimum 8+ years of overall Data Engineering / Software Engineering experience delivering enterprise-scale data solutions.
• Minimum 5+ years of strong hands-on Data Engineering experience, including design, development, optimization, and support of production data pipelines.
• Minimum 3+ years of hands-on Databricks experience, including production experience with:
• Delta Lake
• Medallion Architecture Bronze / Silver / Gold
• Unity Catalog
• Minimum 5+ years of hands-on Spark experience, with strong expertise in PySpark specifically, rather than Scala-only Spark development.
• Minimum 3+ years of AWS data-platform experience, using services such as S3, Glue, Lambda, EMR, Redshift, Kinesis, or related AWS technologies.
• Minimum 2+ years of experience leading complex data-engineering initiatives, technical workstreams, or engineering teams while remaining hands-on technically.
• Proven experience combining data engineering and software-engineering practices, including modular development, testing, source control, CI/CD, observability, and production support.
• Demonstrated daily hands-on use of GitHub Copilot and/or Claude Code CLI, with the ability to describe concrete examples of how these tools are used to accelerate engineering work.
• Demonstrated ability to build POCs and MVPs rapidly using AI-assisted development.
• Strong understanding of distributed data-processing principles, large-scale pipeline design, performance optimization, data quality, and production operations.
• Strong SQL skills and experience working with large enterprise datasets.
• Critical Screening Requirements:
• Candidates should be screened out if they do not demonstrate:
• Proven experience spanning Data Engineering and Software Engineering.
• Proven experience leading complex data-engineering initiatives or teams.
• Hands-on Databricks experience including Delta Lake, Bronze/Silver/Gold Medallion Architecture, and Unity Catalog.
• High-volume Spark and PySpark experience; Scala-only Spark experience is not sufficient.
• Strong experience with the AWS data ecosystem.
• Daily hands-on use of GitHub Copilot and/or Claude Code CLI with specific examples.
• Ability to use AI-assisted development to rapidly create POCs and MVPs.

Nice to Have (But Not a Must):

• 2+ years of experience applying AI/Generative AI within data-engineering solutions, particularly:
• AI-assisted schema inference
• Automated data-quality rules
• LLM-based anomaly detection
• AI-generated PySpark transformations
• Experience building AI-enabled data products, including feature stores, vector stores, RAG data layers, and embedding pipelines.
• 2+ years of high-volume streaming experience using Spark Structured Streaming, Kafka, Kinesis, Auto Loader, or equivalent technologies.
• Strong implementation experience with modern lakehouse patterns including:
• SCD Type 2
• CDC
• Idempotent writes
• Schema evolution
• Data contracts
• Hands-on experience with dbt, Airflow, Databricks Workflows, and/or Delta Live Tables.
• Experience designing data platforms and pipelines supporting enterprise-scale GenAI, RAG, machine-learning, or agentic-AI solutions.