Lead AI Engineer

Job Title : Lead AI Engineer Location: Austin, Texas (Hybrid) Duration: Longterm Contract Lead AI Engineer (Search Modernization) Mandatory Skills: Elastic Search, OpenSearch, Python, LLM, GenAI, Semantic Search, Re-Ranking, AWS, Search Engineer Job Description: We are looking for an AI Engineer to modernize and enhance our existing regex/keyword-based Elastic Search system by integrating state-of-the-art semantic search, dense retrieval, and LLM-powered ranking techniques. This role will drive the transformation of traditional search into an intelligent, context-aware, personalized, and high-precision search experience . The ideal candidate has hands-on experience with Elastic Search internals , information retrieval (IR) , embedding-based search , BM25 , re-ranking , LLM-based retrieval pipelines , and AWS cloud deployment . Roles & Responsibilities Modernizing the Search Platform Analyze limitations in current regex & keyword-only search implementation on ElasticSearch. Enhance search relevance using: BM25 tuning Synonyms, analyzers, custom tokenizers Boosting strategies and scoring optimization Introduce semantic / vector-based search using dense embeddings. 2. LLM-Driven Search & RAG Integration Implement LLM-powered search workflows including: Query rewriting and expansion Embedding generation (OpenAI, Cohere, Sentence Transformers, etc.) Hybrid retrieval (BM25 + vector search) Re-ranking using cross-encoders or LLM evaluators Build RAG (Retrieval Augmented Generation) flows using ElasticSearch vectors, OpenSearch, or AWS-native tools. 3. Search Infrastructure Engineering Build and optimize search APIs for latency, relevance, and throughput. Design scalable pipelines for: Indexing structured and unstructured text Maintaining embedding stores Real-time incremental updates Implement caching, failover, and search monitoring dashboards. 4. AWS Cloud Delivery Deploy and operate solutions on AWS , leveraging: OpenSearch Service or EC2-managed ElasticSearch Lambda, ECS/EKS, API Gateway, SQS/SNS SageMaker for embedding generation or re-ranking models Implement CI/CD for search models and pipelines. 5. Evaluation & Continuous Improvement Develop search evaluation metrics (nDCG, MRR, View email address on click.appcast.io, recall). Conduct A/B experiments to measure improvements. Tune ranking functions and hybrid search scoring. Partner with product teams to refine search behaviors with real usage patterns. Required Skills & Qualifications 5–10 years of experience in AI/ML, NLP, or IR systems , with hands-on search engineering. Strong expertise in ElasticSearch/OpenSearch : analyzers, mappings, scoring, BM25, aggregations, vectors. Experience with semantic search : Embeddings (BERT, SBERT, Llama, GPT-based, Cohere) Vector databases or ES vector fields Approximate nearest neighbor (ANN) techniques Working knowledge of LLM-based retrieval and RAG architectures . Proficient in Python ; familiarity with Java/Scala is a plus. Hands-on AWS experience (OpenSearch, SageMaker, Lambda, ECS/EKS, EC2, S3, IAM). Experience building and deploying APIs using FastAPI/Flask and containerizing with Docker . Familiar with typical IR metrics and search evaluation frameworks. Preferred Skills Knowledge of cross-encoder and bi-encoder architectures for re-ranking. Experience with query understanding , spell correction, autocorrect, and autocomplete features. Exposure to LLMOps / MLOps in search use cases. Understanding of multi-modal search (text + images) is a plus. Experience with knowledge graphs or metadata-aware search. #J-18808-Ljbffr Software Technology, Inc.