Senior Software Engineer - AI Platform

Job Description

Location: New York, NY

\n

\n

We are seeking a Senior Software Engineer to help build the foundational platforms that power enterprise AI products used by legal, tax, and government professionals.

\n

As a member of the AI Platform Engineering team, you will design, build, and operate cloud-native platforms, APIs, developer tools, and infrastructure that enable teams to deliver reliable, compliant, and scalable AI solutions.

\n

\n

This role focuses heavily on the reliability and resilience of mission-critical AI platforms. You will design systems that can withstand provider and dependency failures, establish observability and service-level objectives, and keep the platform secure and current as the AI ecosystem evolves.

\n

The platform includes Kubernetes-based platform-as-a-service frameworks, model-agnostic AI/LLM gateways, hybrid networking, observability, and self-service developer tooling. We are looking for a collaborative and self-motivated engineer who is comfortable navigating ambiguity, takes ownership, and can independently lead complex technical initiatives.

\n

\n

Key Responsibilities

\n

\n
• Improve platform reliability and resilience by designing for failure, defining and meeting SLOs, leading incident response, and reducing operational toil.
\n
• Design, build, and operate Kubernetes-based PaaS frameworks, AI/LLM gateways, APIs, and self-service tools for enterprise AI applications.
\n
• Develop model-agnostic gateway capabilities for providers such as OpenAI, Anthropic, Gemini, and AWS Bedrock.
\n
• Implement intelligent routing, fallback mechanisms, retries, rate limiting, caching, and cost controls across multiple AI providers.
\n
• Build observability solutions covering metrics, logs, distributed traces, dashboards, alerting, and proactive issue detection.
\n
• Develop secure networking solutions connecting applications across public-cloud and on-premises environments.
\n
• Provision and manage AWS infrastructure using Terraform and modern infrastructure-as-code practices.
\n
• Maintain platform security and stability through dependency patching, runtime upgrades, migrations, and provider-integration updates.
\n
• Create reusable frameworks, templates, tools, and workflows that improve developer productivity and reduce operational overhead.
\n
• Evaluate emerging AI technologies and adapt the platform to support new development patterns and business use cases.
\n
• Independently lead large technical initiatives from initial concept through production implementation.
\n

\n

\n

Required Qualifications

\n

\n
• At least six years of professional software engineering experience.
\n
• Strong Python development skills, including experience building production-grade backend services and APIs.
\n
• Experience designing and operating distributed systems in public-cloud environments.
\n
• Strong understanding of system failure modes, fault tolerance, high availability, and resilient design patterns.
\n
• Hands-on AWS experience, including EC2, S3, IAM, and container-based workloads.
\n
• Experience with Infrastructure as Code, preferably Terraform.
\n
• Experience supporting production systems using metrics, logging, tracing, dashboards, alerting, SLOs, and incident-response practices.
\n
• Strong knowledge of software architecture, databases, networking, cloud infrastructure, and modern application development.
\n
• Demonstrated ability to take ownership and independently lead complex technical projects.
\n
• Bachelor’s degree in computer science, engineering, or a related field, or equivalent practical experience.
\n

\n

\n

Preferred Qualifications

\n

\n
• Java development experience.
\n
• Experience building or operating API gateways, LLM gateways, proxy services, or similar platform layers.
\n
• Experience implementing routing, provider fallback, retries, rate limiting, caching, and usage or cost tracking.
\n
• Experience with Kubernetes, preferably Amazon EKS.
\n
• Familiarity with Kubernetes autoscaling technologies such as Karpenter.
\n
• Experience with observability tools such as OpenTelemetry, Prometheus, Grafana, or Datadog.
\n
• Knowledge of AWS networking and security, including VPC, Direct Connect, IAM, and cloud-security controls.
\n
• Experience with chaos engineering, load testing, failure testing, capacity planning, or disaster recovery.
\n
• Experience developing AI-powered applications, agent-based systems, model-inference services, or AI-serving platforms.
\n
• Working knowledge of machine-learning concepts and the ML development lifecycle.
\n
• Experience with AWS SageMaker, AWS Bedrock, PyTorch, TensorFlow, or scikit-learn.
\n
• Familiarity with AI-assisted development tools such as Claude Code, Cursor, or GitHub Copilot.
\n

\n

\n

Ideal Candidate

\n

\n
• The ideal candidate is a senior, hands-on engineer who combines strong Python development skills with deep experience in cloud infrastructure, distributed systems, Kubernetes, and production reliability. You should be comfortable solving complex platform challenges, working independently, and taking large-scale technical initiatives from architecture and design through production deployment.
\n

\n