Principal Software Development Engineer

Principal Software Engineer

We're building a world of health around every individual — shaping a more connected, convenient and compassionate health experience. At CVS Health®, you'll be surrounded by passionate colleagues who care deeply, innovate with purpose, hold ourselves accountable and prioritize safety and quality in everything we do. Join us and be part of something bigger – helping to simplify health care one person, one family and one community at a time.

Job Summary

We are seeking a strategic, hands-on Principal Software Engineer to drive the modernization of our Sales & Acquisition ecosystem, with a strong focus on AI-enabled, cloud-native platforms. This role will lead the design and delivery of a scalable AI-first architecture that combines traditional cloud services with AI Agent orchestration, LLM gateways, MCP servers, and domain-driven APIs.

The Principal Software Engineer will play a pivotal role in accelerating direct-to-consumer (DTC) growth, enhancing lead conversion, and enabling omnichannel engagement—while ensuring enterprise-grade governance, security, compliance (HIPAA/CMS), and performance at scale.

This role owns the technical vision for modern engineering practices, reusable platform services, and intelligent automation using AI agents and governed AI tooling, aligned with enterprise architecture standards.

Key Responsibilities

Architecture & Platform Leadership

• Lead the architectural vision and system design for the engineering organization using a modern, cloud-native and AI-enabled technology stack, with a core focus on reusability, maintainability, scalability, reliability, and performance at scale.
• Define reference architectures and design patterns for AI-powered platforms, including separation of reasoning (AI agents), tools (MCP), models (LLMs), and transactional domain APIs.
• Conduct architecture and design reviews, ensuring alignment with enterprise standards, security, and compliance requirements.

Domain APIs & Microservices

• Design and implement domain-aligned APIs (e.g., Lead, Enrollment, Plan, Consent, Eligibility) using domain-driven design principles.
• Develop API guidelines (naming, versioning, documentation using Open API/Swagger) and manage the full API lifecycle.
• Ensure clean separation between AI reasoning paths and transactional workflows.
• Build secure, scalable APIs and backend services in a HIPAA-regulated environment.

Cloud-Native Engineering (AWS)

• Design and implement cloud-native applications and shared services on AWS, leveraging services such as:
• ECS/Fargate, Lambda, S3, SQS, CloudFront, ALB
• Amazon Connect, AWS Glue, API Gateway
• Ensure solutions meet high availability, resilience, and performance requirements.
• Integrate AWS services and third-party APIs to create seamless, event-driven workflows.

Full‑Stack Engineering

• Implement back-end services using Java, Spring Boot, Python, or other modern languages within a microservices architecture.
• Design, develop, and maintain scalable web applications using React and Angular.
• Work with both RDBMS and NoSQL databases to design and optimize data storage and access patterns.

AI Agent Orchestration & Intelligent Systems

• Design and implement AI Agent orchestration layers that:
• Interpret user intent
• Plan multi-step workflows
• Invoke tools, domain APIs, and models in a controlled manner
• Build enterprise AI agents for use cases such as customer guidance, plan comparison, lead qualification, policy interpretation, and assisted sales.
• Implement tool-based AI reasoning using MCP servers instead of direct data or system access.
• Design multi-agent and fallback strategies to improve resilience, accuracy, and explainability.

MCP Servers (Model & Tool Control Plane)

• Design and build MCP servers as a control plane between AI agents and enterprise tools.
• Enforce governance for:
• Tool registration and access
• Input/output validation
• Context injection
• Auditing and observability
• Integrate MCP servers with:
• Domain APIs
• Document retrieval / RAG services
• Policy and rules engines
• Eligibility and calculation services

LLM Gateway & Model Governance

• Design and implement LLM Gateway layers to centrally manage and govern all LLM access.
• Responsibilities include:
• Model routing and selection
• Prompt templates and standardization
• Token limits, rate limiting, and cost controls
• HIPAA and CMS guardrails
• Integrate with general-purpose LLMs and domain fine-tuned models, ensuring grounded and compliant outputs.
• Prevent direct model access from applications by enforcing gateway-only usage.

DevOps, CI/CD & Observability

• Own the full lifecycle of applications—from architecture and implementation to CI/CD, deployment, and monitoring.
• Help define CI/CD pipelines using GitHub Actions, infrastructure-as-code standards, and observability practices.
• Collaborate with DevOps and infrastructure teams to maintain development, testing, and production environments.

Data & Event‑Driven Architecture

Engineering Excellence & Collaboration

• Improve engineering processes, including coding best practices, Agile delivery, CI/CD, DevSecOps, and quality initiatives.
• Participate in code reviews, providing constructive feedback and ensuring adherence to best practices.
• Coordinate with product, UX, and business teams to align technical delivery with business outcomes.
• Leverage Cursor AI, MCPs, and other AI-assisted development tools for productivity and code reuse.
• Stay current with emerging technologies, especially in AI, agent platforms, and cloud-native architecture.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 10+ years of progressive software engineering experience, with 5+ years in a Principal/Staff Engineer or Architecture-focused role.
• Strong experience designing distributed systems, cloud platforms, and microservices.
• Hands-on experience with AI agents, orchestration patterns, LLM integration, or RAG-based systems.
• Proficiency in front-end technologies such as React and Angular.
• Strong back-end development experience using Java, Python, Spring Boot, or similar modern frameworks.
• Extensive experience with AWS cloud services, including Fargate, Lambda, S3, SQS, CloudFront, and ALB.
• Strong knowledge of CI/CD practices and experience with GitHub Actions.
• Experience with RESTful APIs, OpenAPI/Swagger, and API lifecycle management.
• Experience with both RDBMS (MySQL, PostgreSQL) and NoSQL databases (MongoDB, DynamoDB).
• Exposure to cloud data architectures, analytics, and event-driven systems.

Preferred Qualifications

• Proven ability to influence architectural decisions without formal authority.
• Experience designing or operating AI Agent platforms, MCP servers, or LLM gateways.
• Background in healthcare, Medicare, sales, acquisition, or lead management platforms.
• Experience with customer data platforms (CDPs), data lakes, and real-time analytics.
• Strong understanding of enterprise integration patterns and event-driven architecture.
• Demonstrated experience building governed, compliant AI solutions in regulated environments.

Pay Range

The typical pay range for this role is:

$144,200.00 - $288,400.00

This pay range represents the base hourly rate or base annual full-time salary for