Senior Applied AI Engineer

Senior Applied AI Engineer

Document Intelligence, Retrieval & AI Evaluation

Experience: Senior Level

Position Type: Individual Contributor

Core Technologies: Python, Multimodal LLMs, Hybrid Retrieval, AI Evaluation, Cloud AI

About the Opportunity

We are seeking a highly experienced Senior Applied AI Engineer to own and enhance the applied AI and data science capabilities of a production document-intelligence platform.

The platform processes high volumes of unstructured documents using multimodal large language models, hybrid retrieval systems, and human-in-the-loop review workflows.

The selected engineer will take end-to-end ownership of extraction modeling, retrieval quality, and evaluation infrastructure.

This is a hands-on production engineering role, not a research or proof-of-concept position.

Key Responsibilities

1. Multimodal Document Intelligence

• Design and optimize production-scale multimodal LLM document extraction pipelines.
• Implement schema-constrained decoding, tool calling, structured JSON outputs, and automated validation and repair.
• Develop long-document chunking and stitching strategies.
• Implement field-level confidence scoring and probability calibration.
• Define decision thresholds based on business impact and error costs.

2. Hybrid Retrieval & Search Optimization

• Build and optimize dense, learned-sparse, and lexical retrieval systems.
• Design retrieval strategies for large reference catalogs.
• Evaluate approximate nearest-neighbor indexing approaches.
• Implement rank fusion and reranking techniques.
• Optimize retrieval quality against latency and infrastructure costs.

3. AI Evaluation & Experimentation — Critical Requirement

• Own the design and implementation of evaluation harnesses for non-deterministic AI systems.
• Develop labeled datasets, annotation standards, and reproducible evaluation pipelines.
• Apply statistical hypothesis testing, confidence intervals, and appropriate sample sizing.
• Evaluate precision, recall, F1, field-level accuracy, and end-to-end system performance.
• Conduct detailed error analysis and identify performance gaps across document types and customer populations.
• Establish automated regression checks and production release quality gates.
• Use evaluation evidence to guide architectural and model selection decisions.

4. Production AI Engineering

• Develop reliable Python-based AI services and processing pipelines.
• Implement asynchronous processing, typed application models, and robust error handling.
• Build event-driven processing services with idempotency, deduplication, retries, and dead-letter handling.
• Implement safe deployment, rollback, replay, and backfill mechanisms.
• Integrate experiment tracking, observability, and automated CI/CD practices.

5. Prompt Engineering & AI Lifecycle Management

• Treat prompts as version-controlled engineering artifacts.
• Implement prompt testing, review, and automated validation.
• Support document-class and tenant-specific prompt configurations.
• Track changes to prompts, models, datasets, and evaluation results.

6. Security, Privacy & Regulatory Compliance

• Implement data classification, minimization, and redaction.
• Enforce tenant isolation across data storage, search indexes, prompts, and telemetry.
• Implement retention policies, least-privilege access, and secure secrets management.
• Ensure AI processing aligns with applicable regulatory requirements.

Required Qualifications

• 8+ years of hands-on production Python experience , including operating production services.
• Proven experience deploying and supporting production AI/ML systems.
• Strong expertise in multimodal LLM-based document extraction.
• Hands-on experience with production hybrid retrieval systems.
• Demonstrated ownership of AI/ML evaluation frameworks.
• Strong statistical foundations, including hypothesis testing, confidence intervals, sampling, and calibration.
• Experience with dataset creation, annotation protocols, and error analysis.
• Advanced Python proficiency, including Asyncio, type annotations, Pydantic v2, packaging, and profiling.
• Experience with event-driven architectures and reliable distributed processing.
• Working knowledge of data privacy, security, and multi-tenant architectures.

Mandatory: Candidates must demonstrate that they personally built an evaluation harness for a non-deterministic system and used its results to make an important engineering or production decision.

Preferred Qualifications

• Production experience with agentic AI orchestration.
• Strong understanding of LLM economics, token optimization, caching, and cost-per-document analysis.
• Experience with A/B testing, shadow deployments, and online experimentation.
• Experience implementing drift detection and production model monitoring.
• Knowledge of LLM-as-judge methodologies and their limitations.
• Experience with fine-tuning, adapters, distillation, or systematic prompt optimization.
• Experience with large-scale analytical data platforms.
• Familiarity with FERPA, HIPAA, GDPR, PCI-DSS, or SOX environments.

Technology Environment

Programming: Python 3.12, Asyncio, Pydantic v2

AI/ML: Commercial multimodal LLMs, structured extraction, agent orchestration, experiment tracking

Search: Dense and sparse embeddings, ANN vector search, hybrid retrieval, reranking

Cloud: Managed cloud AI services, serverless containerized workloads, event-driven messaging

DevOps & Monitoring: GitLab CI/CD, OpenTelemetry, automated testing, production monitoring

Experience with another major cloud platform is acceptable where the candidate can demonstrate equivalent architectural knowledge.

Additional Technical Skills

Experience with Go, Rust, TypeScript, React, Kubernetes, or infrastructure as code is welcome but not required .

These technologies are not screening criteria.

Who Should Apply?

We're looking for engineers who:

• Have personally built, deployed, and operated AI systems in production.
• Understand how to measure and improve AI reliability.
• Can defend architectural decisions using evaluation evidence.
• Combine strong Python engineering skills with applied machine learning and statistical expertise.
• Are comfortable owning complex systems from modeling through production operations.

This position is best suited for a senior hands-on engineer with demonstrated production ownership, not someone whose experience is primarily limited to prototypes or notebook experimentation.

How to Apply

Please submit your updated resume and include a brief example of an AI evaluation framework or production AI system you have personally designed and implemented.