Generative AI Engineer

Position : Generative AI Engineer



Location : Los Angeles, CA Onsite

Duration : Long term contract



Specialization : Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation

Experience : 6 8+ Years

Job Summary

We are seeking an experienced AI Engineer with strong hands-on expertise in building, deploying, and monitoring production-grade AI agents and Generative AI solutions using AWS technologies.

The ideal candidate will have experience developing AI agents using Claude Agent SDK , deploying them on Amazon Bedrock AgentCore Runtime , automating deployments through GitHub Actions , and implementing Langfuse for observability and continuous evaluation of agent traffic .

The role requires a strong combination of AI engineering, cloud deployment, data pipeline development, and LLM performance monitoring.

Key Responsibilities

1. Agentic AI Development

• Design, develop, and deploy AI agents using Claude Agent SDK and Python.
• Build intelligent, autonomous agents capable of executing complex workflows and interacting with external tools, APIs, and knowledge bases.
• Develop agent orchestration workflows with appropriate error handling, retry mechanisms, and guardrails.
• Integrate AI agents with enterprise applications and data sources.

2. AWS Bedrock & AgentCore

• Deploy and manage AI agents using Amazon Bedrock AgentCore Runtime.
• Develop scalable, secure, and productionlos -ready agent architectures on AWS.
• Integrate agents with Amazon Bedrock foundation models and enterprise knowledge bases.
• Implement agent authentication, access controls, logging, and runtime monitoring.

3. Langfuse Deployment & AI Observability

• Deploy, configure, and maintain Langfuse for comprehensive LLM and agent observability.
• Integrate Langfuse with AI agents to capture traces, prompts, responses, tool calls, and execution metrics.
• Monitor agent performance, response latency, token consumption, and operational costs.
• Identify and troubleshoot production issues through distributed tracing and monitoring dashboards.

4. Continuous Agent Evaluation

• Design and implement continuous evaluation frameworks for production AI agent traffic.
• Evaluate agent responses for accuracy, relevance, hallucinations, task completion, and tool execution quality.
• Develop automated evaluation pipelines using Langfuse, LLM-as-a-Judge, and custom evaluation metrics.
• Implement continuous feedback mechanisms to improve agent reliability and effectiveness.
• Establish quality benchmarks and detect agent performance regressions.

5. Data Pipelines & Knowledge Base Integration

• Develop AWS Step Functions workflows integrating Amazon Textract and Amazon Comprehend for intelligent document processing.
• Build document ingestion, preprocessing, and data transformation pipelines.
• Implement vector database solutions, including document chunking, embedding generation, and vector indexing.
• Integrate AI agents with Retrieval-Augmented Generation (RAG) pipelines and enterprise knowledge bases.
• Optimize retrieval performance and contextual accuracy.

6. CI/CD & Deployment Automation

• Build and maintain CI/CD pipelines using GitHub Actions.
• Automate agent code builds, testing, deployment, and configuration management.
• Implement deployment workflows for Amazon Bedrock AgentCore Runtime.
• Integrate automated agent evaluations into CI/CD processes.
• Apply DevOps best practices for version control, release management, and production monitoring.

Required Technical Skills

• Programming: Strong Python development experience.
• Agentic AI: Claude Agent SDK, AI agent development, agent orchestration, tool calling.
• AWS AI Services: Amazon Bedrock, Bedrock AgentCore Runtime, Step Functions, Textract, Comprehend.
• Observability: Langfuse deployment, LLM tracing, monitoring, performance analysis.
• Evaluation: Continuous agent evaluation, LLM-as-a-Judge, evaluation metrics and automated testing.
• DevOps: GitHub Actions, CI/CD pipelines, Git, deployment automation.
• RAG: Vector databases, embeddings, chunking strategies, indexing, knowledge base integrations.
• Cloud: AWS architecture, IAM, CloudWatch, security, and deployment practices.

Good-to-Have Skills

• Experience with LangChain, LangGraph, or other agent orchestration frameworks.
• Experience with RAGAS, DeepEval, or similar LLM evaluation frameworks.
• Familiarity with Amazon OpenSearch, Pinecone, FAISS, or pgvector.
• Experience with Docker, Terraform, or infrastructure as code.
• Knowledge of Model Context Protocol (MCP).
• Experience with prompt engineering, guardrails, and AI governance.
• Experience deploying enterprise Generative AI solutions in production environments.

Qualifications

• 6 8+ years of software engineering, cloud engineering, or AI/ML development experience.
• Strong hands-on experience building Generative AI or Agentic AI applications.
• Demonstrated experience deploying and supporting AI agents in production.
• Strong understanding of AWS cloud services, LLM architectures, and automated deployment processes.
• Ability to independently design, implement, troubleshoot, and optimize AI agent solutions.