AI Data Engineer

Title: AI Data Engineer







Location: Remote







Contract: 12 Months contract















Typical Day in the Role

Purpose of the Team: This role works in a people-centric, culture-first environment alongside designers, product partners, engineers, strategists, and writers to design and build modern experiences

that make intelligent systems feel clear, trustworthy, and human.

The team are a forward-looking and passionate about building innovative technology and shipping delightful experiences-especially where AI changes what products can do. We invest in rapid prototyping, model tuning and evaluation, data experiments, and AI-related explorations to help explore the future of intelligent Microsoft products. The team brings diverse backgrounds in multiple disciplines, from engineering, design, or research to behavioral sciences or medicine; we come in all shapes and sizes. We love pushing boundaries and helping evolve Microsoft's products. Our current focus is on Copilot in the M365 Enterprise world.

Role overview

Team are seeking a hands-on senior backend engineer with deep expertise in data generation and processing, scalable pipelines, large-scale storage, and quantitative analysis. The ideal candidate combines strong software engineering fundamentals with practical experience building and maintaining reliable, cloud-based backend systems.

The candidate will take ownership of technical work from assessment and design through implementation and ongoing support. This role requires sound engineering judgment, comfort working in existing codebases, and the ability to translate evolving requirements into maintainable solutions while collaborating closely with engineering and cross-functional partners.

About you

You're a self-driven backend engineer who cares about building reliable, scalable foundations for thoughtful product experiences. You can turn evolving ideas into maintainable services and data workflows, learn new tools quickly, and are comfortable exploring the unknown. You value strong engineering practices and know how to collaborate across engineering, design, and product.

You contribute openly, prototype or experiment to learn, share feedback generously, and help raise the quality of the team's work. You are curious, adaptable, and motivated to build systems that make a real impact.

Key responsibilities

• Design, build, and evolve synthetic data-generation systems, assessing existing approaches and technical trade-offs while developing scalable, maintainable workflows that translate structured inputs, persona and organizational context into coherent, traceable, and reproducible datasets.
• Build and operate scalable pipelines and orchestration, applying concurrency, asynchronous execution, and resilient failure handling where appropriate.
• Design and evolve data models, storage approaches, and retrieval patterns to support large data volumes and efficient access.
• Develop and evolve validation frameworks and systems that define quality criteria, automate checks, and use quantitative analysis and profiling to assess data quality, investigate system behavior, and drive improvements in performance and reliability.
• Apply cloud engineering practices to build, deploy, monitor, and support dependable systems, balancing scalability, operational simplicity, and cost.
• Evaluate technical trade-offs, communicate progress and risks, and collaborate on practical solutions. Maintain clear documentation, automated tests, and guidance that support ongoing ownership.

Candidate Requirements

Disqualifiers:

Missing the core required technical qualifications listed in the JD, including the required software/data engineering background, data pipelines/ETL, and LLM/AI systems experience.

Preferred skills and experience

• Experience with Python
• Experience integrating LLMs through AI platforms, frameworks, and APIs, including building or evaluating production-oriented AI workflows.
• Experience designing and running evaluations of LLM output, such as defining quality criteria, building automated checks.
• Experience with distributed processing, streaming or event-driven systems, and workflow orchestration tools-such as Apache Spark, Kafka, or equivalent technologies-and familiarity with relational or NoSQL databases, cloud object storage, or data lake architectures
• Experience with observability, performance tuning, and deployment automation is also valuable; no specific cloud provider or storage product is required
• Prior Microsoft experience is only a nice-to-have, not required
• Experience with Apache Spark, Kafka, or equivalent distributed-processing and event-driven technologies.
• Familiarity with relational or NoSQL databases, cloud object storage, and data lake architectures.
• Ability to work independently within an existing codebase and translate evolving requirements into maintainable solutions.
• Strong technical communication, documentation, collaboration, adaptability, and problem-solving skills.
• Experience creating automated tests and data-quality validation frameworks
• Portfolio/work samples are optional

Required skills and experience

• 5+ years of software or data engineering experience, or equivalent significant experience building systems with coding in languages & frameworks including, but not limited to, Python, Spring Boot, .NET, with demonstrated ownership through implementation and support.
• Strong backend programming and systems engineering skills, including designing maintainable services, modular components, APIs, and automated tests.
• Deep experience designing, building, and operating data generation, ETL, and distributed data-processing workflows, including creating, scaling, and maintaining cloud-based pipelines.
• Strong working understanding of LLMs, including their capabilities, limitations, common failure modes, and the role of data quality, prompting, evaluation, and model behavior in AI-powered systems.
• Working knowledge of orchestration, concurrency, asynchronous execution, streaming or event-driven architectures, and recovery from failures.
• Strong knowledge of large-scale data storage, modeling, and retrieval, including partitioning, indexing, access patterns, and trade-offs among storage approaches.
• Strong quantitative analytical and troubleshooting skills, with the ability to validate data and system behavior, interpret performance measurements, and make evidence-based engineering decisions.
• Practical cloud engineering experience and the ability to work independently in an existing codebase. Clear technical communication, effective collaboration, and a disciplined approach to documentation and operational reliability.

Hard Skills Assessments

Expected Dates that Hard Skills Assessments will be scheduled: asap

Hard Skills Assessment Process: The assessment process will include 1-2 rounds of HSA

Required Candidate Preparation: Questions about their previous technical work, including how they designed solutions, evaluated trade-offs, troubleshot issues, and supported systems after implementation.

• Behavioral questions focused on adaptability, problem-solving, out-of-the-box thinking, communication, and working effectively with cross-functional teams.

Additional Information

Explain a typical day in the role.: Hands-on backend/data engineering focused on building and improving our synthetic data-generation system: evolving generation workflows, pipelines and orchestration; debugging and improving existing code; designing data/storage approaches; implementing validation and automated quality checks; and collaborating with engineers, researchers, designers and product partners to translate evolving requirements into reliable systems.

What is the ideal background of a candidate for this role: Senior software or data engineer with 5+ years of experience building production backend/data systems. Strong background in data generation, ETL/distributed processing, cloud pipelines and large-scale data/storage systems, paired with working knowledge of LLMs and AI workflows. Comfortable owning ambiguous engineering problems end-to-end in an existing codebase.

What are unique selling points that would get candidates interested in your role over another: Opportunity to work at the intersection of AI, synthetic data, backend engineering and UX/product experimentation, building infrastructure that directly enables Microsoft's exploration of future M365 Copilot experiences. The work has broader importance for our model-forward prototyping ambitions, rather than being routine pipeline maintenance.

How will contractor performance be measured:

• Delivery and quality of agreed improvements to the synthetic-data generation system.
• Demonstrated improvements in pipeline scalability, reproducibility, resilience, and reliability.
• Delivery of effective automated validation and data-quality measurement.
• Successful end-to-end delivery of assigned technical work, including testing, documentation, deployment, and support.

Top 3 Must-Have HARD Skills & years of experience for each:

1. Software/backend or data engineering: 5+ years
2. Data generation, ETL & distributed/cloud data pipelines: substantial hands-on experience within that 5+ year background
3. LLM/AI systems: strong working understanding plus experience integrating/evaluating LLM-based workflows.