Principal Engineer, Ai & Agentic Systems

BICP, a San Diego-based consulting firm specializing in AI, Data & Analytics, is seeking an experienced Principal Engineer, AI & Agentic Systems to join our team in an embedded capacity with one of our strategic clients in the apparel and retail industry. We will consider remote candidates with the right blend of experience however candidate must be willing to be onsite 1 week a month initially. This is a hands-on and technical leadership role for someone who can establish platform direction, write production code, and turn business needs into reliable AI applications.

You’ll work with engineering and business leaders to identify valuable opportunities, choose the right approach, and deliver solutions that teams can use and trust. The scope includes AI assistants, agent-driven workflows, enterprise knowledge retrieval, and reusable services that support future development.

What You’ll Do

• Own key architectural decisions for the AI platform, including technology selection, build-versus-buy recommendations, and reusable engineering patterns.
• Design and ship AI applications and agentic workflows using LLMs, tool calling, planning, memory, and human review where appropriate.
• Connect AI solutions to enterprise APIs, databases, SaaS applications, and business systems.
• Develop orchestration and retrieval capabilities that support multiple applications and use cases.
• Establish evaluation, testing, monitoring, and observability practices to measure accuracy, safety, reliability, and business results.
• Incorporate security, access controls, governance, and operational guardrails into solution design.
• Move promising prototypes into production and improve them through usage data and stakeholder feedback.
• Help engineers strengthen their capabilities through technical reviews, pairing, and practical tooling.
• Assess whether a problem calls for an agentic approach, conventional automation, or another solution.

What You Bring

• 10+ years of software engineering experience, including at least 2 years building and operating AI systems in production.
• Hands-on experience deploying agentic systems that use planning, reasoning, memory, and tools to complete business workflows.
• Strong Python skills and experience developing APIs, backend services, and cloud-native applications.
• Experience creating AI evaluation frameworks and production monitoring that assess quality, reliability, safety, and operational performance.
• A strong understanding of LLM platforms and orchestration frameworks, with the ability to evaluate tradeoffs and select technologies based on the problem.
• Experience integrating applications with enterprise data and systems.
• The ability to clarify ambiguous requirements, explain technical decisions, and collaborate effectively with business stakeholders.
• A bachelor’s degree in computer science, engineering, AI, data science, or a related field, or equivalent practical experience.

Additional Experience We Value

• RAG, embeddings, vector databases, and enterprise knowledge retrieval.
• Enterprise AI platforms such as Azure AI Foundry, Google Gemini Enterprise, Microsoft Agent 365, or Copilot Studio.
• Azure, AWS, or Google Cloud.
• AI governance, security, compliance, and observability.
• MLOps, model lifecycle management, and deployment pipelines.
• Retail, apparel, ecommerce, direct-to-consumer, or other consumer businesses.

About BICP

BICP is a San Diego-based AI, Data & Analytics consultancy that helps organizations close the gap between strategic priorities and the specialized capabilities required to execute them. Our Forward Deployed practitioners work alongside client teams, bringing deep expertise across AI/ML, Data, Analytics, Product, and Architecture to move initiatives from idea to implementation, production, and measurable business impact.