Senior Fullstack Data & AI Search Engineer

Job Title - Senior Fullstack Data & AI Search Engineer



Experience Required - 8+ Years



Timezone - Approx 1:00 PM / 2:00 PM and 10:00 PM / 11:00 PM IST (CET Time Zone)
Budget - 1.5 LPM
Work Mode - Remote

Experience: 8+ Years

Profile: Senior Data & AI Search Engineer with hands-on expertise in RAG pipelines and agentic AI workflows



Primary Focus: Elasticsearch + RAG + agentic AI workflows



Experience: Senior (8+ years total; 6+ years in enterprise search / RAG / LLM applications)

Role Overview

We are looking for a hands-on Data & AI Search Engineer to design and deliver a production-grade, AI-augmented enterprise search capability for a large international organisation. The engagement covers the full pipeline from raw data ingestion through to AI-generated, grounded answers surfaced via a conversational or search interface.

The right candidate combines deep Elasticsearch engineering with practical experience building Retrieval-Augmented Generation (RAG) pipelines and agentic AI workflows . This is an individual contributor role with direct impact on a critical knowledge management platform.

Key Responsibilities 1. Data Engineering and Ingestion

Design and build scalable ingestion pipelines and connectors from enterprise sources including SharePoint, Liferay, web crawls, Data Lakes, and corporate systems into Elasticsearch or equivalent search indexes .
Support batch, incremental, and near-real-time indexing ; implement change tracking, version management, source provenance, access permission mapping, and deletion event handling to keep the index accurate.
Build document conversion pipelines for PDF, Word, Excel, PowerPoint, HTML, email, and scanned content ; convert to structured Markdown and vector embeddings using tools such as Marker, Docling, or equivalent frameworks .
Design semantic chunking strategies (chunk size, overlap, section-aware splitting, heading preservation, table handling) and implement metadata extraction, enrichment, and deduplication during ingestion.

2. Retrieval and Search

Develop hybrid search capabilities combining BM25 keyword search, semantic vector search, metadata filtering, and contextual retrieval .
Build re-ranking pipelines using embedding models, cross-encoders, or custom ranking logic to improve result relevance.
Implement advanced retrieval techniques: query rewriting, query expansion, multi-query retrieval, parent-child retrieval, contextual document embeddings, and contextual compression .
Enforce security controls so users retrieve only content they are authorised to access .

3. RAG Pipeline and Agentic Workflows

Design and build the end-to-end RAG pipeline connecting enterprise search to large language models for grounded answer generation.
Implement agentic workflows where the AI can invoke tools, call enterprise APIs, perform multi-step reasoning, and refine searches iteratively to answer complex queries.
Engineer prompt orchestration patterns: system prompts, retrieval prompts, guardrails, context assembly, response formatting, and fallback strategies for low-confidence or ambiguous queries .

Technical Requirements Core Search Engineering

Deep, hands-on Elasticsearch experience: query DSL, BM25 tuning, function_score, boosting and decay functions, multi-field matching .



Index and data modelling: field type selection, custom analyzers and tokenizers per content type (code, prose, structured records, multimedia).



Cluster operations: shard strategy, index sizing, reindexing, query latency tuning, and cluster health management .



Search evaluation and relevance testing: building ground-truth benchmarks, measuring precision/recall, NDCG , and iterating against them.
Experience with Elasticsearch, OpenSearch, Azure AI Search, or equivalent enterprise search platforms .

Data and Ingestion Engineering

Proven experience building or configuring connectors for SharePoint, Liferay, databases, and Azure Data Lake including incremental sync, CDC, rate limiting, and API edge-case handling .



Proficiency in Python ; experience with data processing frameworks and document conversion libraries.

AI and RAG Engineering

Hands-on experience with embedding models, re-ranking models, cross-encoders, prompt engineering, and response grounding techniques .
Experience with LLM orchestration frameworks: LangChain, LlamaIndex, Haystack, or equivalent .
Practical experience with tool calling, agentic workflows, function calling, and multi-step retrieval .
Experience integrating with commercial or open-source LLMs: Azure OpenAI, OpenAI, Anthropic, Google Gemini, Meta Llama, Mistral, or similar .

Frontend

Working knowledge of React or equivalent front-end technologies to support search UI integration (desirable, not mandatory).

Qualifications and Experience

First-level university degree in Computer Science, Computer Engineering, Information Systems , or a related discipline.



8 years of professional experience in software or data engineering .
Minimum 6 years of hands-on experience building enterprise search, AI-powered search, semantic search, RAG, or LLM-based applications .



Excellent written and verbal communication skills in English .

What This Engagement Offers

We have built a high-visibility knowledge management platform for a large international organisation.
End-to-end ownership across data engineering, retrieval, and GenAI layers.
Fully remote, flexible working arrangement within agreed time zone coverage.
Potential for contract extension based on delivery and business need.

While sharing the profile of your resource, share the details below, too

	Technology / Skill Area	Candidate Experience (Years)	Self Rating (Out of 10)
1	Overall Years of Experience
2	Relevant Experience
3	Overall Software / Data Engineering Experience		/10
4	Enterprise Search / AI Search / Semantic Search		/10
5	Elasticsearch		/10
6	Elasticsearch Query DSL, BM25, Boosting, function_score		/10
7	Elasticsearch Index & Data Modelling		/10
8	Elasticsearch Analyzers, Tokenizers & Search Relevance		/10
9	Elasticsearch Cluster Management, Sharding & Performance Tuning		/10
10	Python & Data Processing		/10
11	Enterprise Data Ingestion / Connectors		/10
12	SharePoint / Liferay / Database / Azure Data Lake Integration		/10
13	Document Processing PDF, Word, Excel, PPT, HTML, Email, OCR		/10
14	Semantic Chunking / Metadata Extraction / Deduplication		/10
15	Vector Search / Embeddings / Hybrid Search		/10
16	RAG Pipeline Design & Implementation		/10
17	Re-ranking / Cross-Encoders / Retrieval Optimization		/10
18	LangChain / LlamaIndex / Haystack or Equivalent		/10
19	Prompt Engineering / Context Assembly / Grounding / Guardrails		/10
20	Agentic AI / Tool Calling / Function Calling		/10