Senior AI Engineer

What We Do: Florence software advances cures by helping the world’s most important research sites do their best work. Our solutions are now used by over 30,000 research teams in 70 countries around the world—we’re the most widely deployed site workflow tool in the industry. By the end of the decade, we’ll double the pace at which new medicines get to market by doubling the output of trial site teams. To date, we were named a Deloitte Fast 50 business, G2 Category Leader, an Inc. & AJC best place to work, and an Inc. 5000 company five years in a row. At Florence, we are committed to make the world a better place by accelerating research while providing an environment for our employees where they can be happy in their lives, enjoy their jobs, and grow. What You’ll Bring to The Team:We are looking for a Senior AI Engineer to design, build, and deploy high-quality AI-powered features. This role focuses on owning end-to-end implementation of AI systems within a product area — from prototyping to production — with a strong emphasis on reliability, iteration, and measurable impact. You will work closely with product and engineering teams to turn ambiguous problems into effective AI solutions, while contributing to best practices and raising the bar for AI development.You Will:End-to-End AI Feature OwnershipDesign and implement AI-powered features (LLM workflows, copilots, and agent-based systems with tool use and multi-step reasoning)Own the full lifecycle: prototyping evaluation production deployment iterationEnsure solutions are reliable, performant, and aligned with product needsAI System ImplementationBuild and optimize:Prompt pipelines for specific use casesRetrieval systems (embeddings, chunking, ranking)RAG-based workflows where neededIterate on outputs to improve quality, accuracy, and consistencyDesign scalable and cost-efficient AI architectures for production workloadsSelect and evaluate models (hosted vs open-source) based on use case constraintsAgent-Based Systems (AgentCore)Design and build agentic workflows capable of multi-step reasoning and decision-makingIntegrate agents with tools, APIs, and internal systems to perform real-world actionsImplement planning, execution, and reflection loops for complex tasksManage context, memory, and state across multi-step interactionsBalance deterministic workflows vs. agent autonomy for reliability and controlExperimentation & EvaluationRun structured experiments to compare approaches (prompting, retrieval, models)Define and track key metrics for AI performance (quality, latency, cost)Debug and improve non-deterministic system behaviorBuild and maintain evaluation datasets and benchmarksImplement automated evaluation pipelines for continuous improvementCollaboration & ContributionDrive technical direction and influence AI adoption across teamsPartner with product managers and designers to scope AI featuresContribute to shared patterns and reusable componentsParticipate in code reviews and design discussionsSupport and mentor mid-level engineers where neededAI Reliability, Safety & GovernanceDesign guardrails to ensure safe and reliable AI behaviorMitigate hallucinations, prompt injection, and model misuseEnsure compliance with data privacy and enterprise requirementsImplement monitoring and observability for AI systems in productionImplement guardrails for agent actions (tool access control, execution boundaries)Prevent failure cascades in multi-step agent workflowsAn Ideal Candidate Has: Core AI SkillsStrong understanding of LLM capabilities and limitationsExperience with prompt engineering and structured output designHands-on experience with embeddings and vector searchFamiliarity with RAG architectures and when to apply themExperience designing agent-based architectures (AgentCore concepts)Understanding of tool use, planning strategies, and memory mechanisms in LLM systemsEngineering Skills4+ years of related work experience Solid backend/system design fundamentalsEvaluate agent performance across multi-step tasks (task success rate, error propagation)Debug and optimize agent decision-making and tool selection behaviorExperience building and deploying production-grade systemsAbility to debug complex issues, including probabilistic outputsComfort working with APIs, pipelines, and data flowsProduct ThinkingAbility to translate user needs into effective AI solutionsStrong intuition for balancing quality, latency, and costFocus on delivering measurable product impactCollaborationCommunicates clearly across engineering and product teamsContributes to team knowledge and shared practicesWhat Success Looks Like:Ships high-quality AI features that deliver clear user valueDemonstrates strong ownership from idea to productionImproves systems through structured iteration and experimentationContributes to team-level best practices and reusable solutionsWhat’s in it for you?Do well. We offer a competitive compensation package, medical and dental insurance, and office space in the heart of the city.Do good. We insist that health technology is the highest calling for software development. We pride ourselves on working on something bigger than ourselves; helping advance cures and therapies.Make the leap. Join our high-output culture to create innovative, modern, and purposeful software solutions.Florence supports workplace diversity and does not discriminate on the basis of race, color, religion, gender identity or expression, national origin, age, military service eligibility, veteran status, sexual orientation, marital status, physical disability, or any other protected class.Please be cautious of potential recruitment fraud. If you are interested in exploring opportunities at Florence Healthcare, please go directly to our Careers Page. Florence Healthcare will never ask you to pay a fee or download software as part of the interview process with our company. In addition, Florence Healthcare will not ask for your personal banking information until you have signed an offer of employment and completed onboarding paperwork that is provided by our People Operations team. All communications with Florence Healthcare employees will only be sent from @florencehc.com email addresses.