Lead AI/ML Engineer

About the Opportunity

Our client is a growing technology consultancy focused on helping banks build and deploy secure, production-grade AI solutions.

They are looking for a Lead AI/ML Engineer to help build the intelligence layer behind an AI platform designed specifically for the banking industry.

This is a hands-on engineering role focused on production LLM applications, RAG, retrieval, model serving, tool orchestration, and evaluation. The goal isn't simply to build an impressive AI demo—the goal is to build systems that produce accurate, verifiable, and auditable answers in a highly regulated environment.

You'll work closely with data engineering, platform engineering, and product teams to build AI systems that can operate securely within banking environments, including self-hosted and on-premises deployments.

What You'll Do

• Design and build production-grade RAG and retrieval systems that provide verifiable answers with supporting queries, sources, and citations.
• Develop retrieval and verification workflows across structured and unstructured banking data.
• Deploy and operate self-hosted open-weight LLMs and embedding models, including within individual bank environments.
• Build and evolve the organization's standardized RAG architecture and patterns.
• Design and implement an MCP/tooling layer that allows AI systems to securely interact with governed data, metrics, documents, and customer information.
• Build tool/function-calling workflows with strong auditing, access controls, and clear boundaries around what AI systems can access.
• Develop and own an AI evaluation framework covering golden-question regression testing, retrieval quality, groundedness, answer accuracy, and appropriate "I don't know" behavior.
• Make evaluation a formal release gate for production AI systems.
• Implement LLM guardrails including PII redaction, prompt-injection defenses, context controls, and query/row limits.
• Partner with data engineering to ensure AI systems use governed semantic-layer metrics rather than generating unreliable ad hoc SQL.
• Work with platform and security engineering to deploy reliable AI infrastructure in controlled environments.
• Monitor and optimize latency, inference cost, retrieval quality, and model performance across different models and deployment configurations.
• Document model architecture, evaluation methodology, security controls, and guardrails for customer security reviews and audits.
• Provide technical leadership while remaining deeply hands-on with implementation and production troubleshooting.

What We're Looking For

• 6+ years of software/ML engineering experience, with at least 2–3 years building and shipping production LLM, RAG, NLP, or generative AI systems.
• Proven experience taking AI systems from prototype to production and real users.
• Strong Python skills and solid software engineering fundamentals.
• Hands-on experience building production RAG/retrieval pipelines.
• Strong understanding of LLM evaluation, including groundedness, retrieval quality, regression testing, and hallucination reduction.
• Experience implementing AI guardrails and security controls.
• Experience deploying or operating self-hosted/open-weight models.
• Ability to reason about the trade-offs between model quality, latency, infrastructure, and cost.
• Strong understanding of prompt engineering, tool calling, structured outputs, and LLM application architecture.
• A strong engineering mindset around accuracy and verifiability, rather than simply building compelling AI demos.
• Comfortable working cross-functionally with data, platform, security, and product teams.

Technology Environment

The team works across a modern AI/ML stack including:

Languages: Python

Model Serving: vLLM, Ollama

Infrastructure: NVIDIA GPUs, CUDA, NVIDIA AI Enterprise

RAG & Retrieval: LlamaIndex, Haystack, Qdrant, pgvector

Embeddings: Modern embedding models and vector search

AI Orchestration: MCP, tool/function calling

Structured AI: Text-to-SQL, semantic layers, Cube, dbt MetricFlow

Evaluation: LLM evaluation frameworks, groundedness testing, retrieval evaluation

AI Security: PII redaction, prompt-injection defense, access controls

You don't need experience with every technology listed. We're particularly interested in engineers who understand the architecture and engineering principles behind production AI systems and can quickly adapt to new tools.

Why This Role?

• Build real AI, not prototypes: Your work will be deployed into production and used by banking professionals.
• Own the AI architecture: You'll have meaningful influence over RAG, model serving, evaluation, tooling, and guardrails.
• Work with self-hosted AI: Build and operate LLM infrastructure where security, privacy, and deployment constraints make traditional cloud-only approaches impractical.
• Solve hard reliability problems: Focus on making AI systems accurate, grounded, observable, and resistant to failure.
• AI + Data: Work at the intersection of LLM engineering, structured data, semantic layers, and intelligent retrieval.
• High-impact environment: In banking, an incorrect answer isn't just a poor user experience—it can have significant business and regulatory consequences.
• Technical leadership: Establish engineering standards and best practices for production AI systems while remaining hands-on.

Nice to Have

• Experience building AI systems in banking, financial services, healthcare, cybersecurity, or another high-stakes/regulated environment.
• Experience deploying AI systems in on-premises or highly controlled environments.
• Experience with fine-tuning, adapters, or retrieval-quality optimization.
• Familiarity with banking, financial services, or financial data.
• Experience with AI security, governance, or compliance frameworks.
• Experience working with GPUs, CUDA, and production inference infrastructure.

The Ideal Candidate

This role is particularly well suited for someone who has moved beyond experimentation with LLMs and has actually operated AI systems in production.

You might be a Lead AI Engineer, Senior ML Engineer, LLM Engineer, Applied AI Engineer, or Staff AI Engineer who has built RAG systems, deployed models, implemented evaluation frameworks, and dealt with the messy realities of production AI.

The strongest candidates will be engineers who are obsessed with the question:

"How do we know the AI is actually right?"

If you're excited by building reliable AI systems where accuracy, security, and verifiability matter as much as model performance, we'd love to hear from you.