Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Sr. Manager, Engineering - AI Gateway (LLM Inference)

$228.6k - $314.25k

Menlo Ventures

At Databricks, we are passionate about enabling data teams to solve the world’s toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world’s best data and AI infrastructure platform so our customers can use deep data insights to improve their business. Founded by engineers — and customer obsessed — we leap at every opportunity to tackle technical challenges, from designing next-gen UI/UX for interfacing with data to scaling our services and infrastructure across millions of virtual machines. And we're only getting started. The impact you will have As the Sr. Manager of Engineering in our newly opened NYC engineering hub, you will be leading teams building multiple products that enable our customers to use innovative ML/AI techniques to add value to their business. You will work closely with product and research on cutting‑edge technologies and be responsible for launching and growing new products, playing a key role in defining the future of how AI and data applications are built. The Databricks AI Gateway is an enterprise control plane for governing, routing, and monitoring LLM endpoints, coding agents, and model serving endpoints on Databricks. AI Gateway lets you standardize, secure, and observe all LLM inference traffic on Databricks, while capturing detailed production data to manage cost, performance, and quality over time. What we look for 8+ years of industry experience building and supporting large-scale data or production applications Experience building classic ML or GenAI systems. A passion for building and scaling early‑stage products Ability to adapt in a fast paced environment. Motivated by delivering customer value and impact. Experience driving company initiatives towards customer satisfaction. BS/MS/PhD in Computer Science or related majors, or equivalent experience. Pay Range Transparency Databricks is committed to fair and equitable compensation practices. The pay range(s) for this role is listed below and represents the expected salary range for non-commissionable roles or on-target earnings for commissionable roles. Actual compensation packages are based on several factors that are unique to each candidate, including but not limited to job-related skills, depth of experience, relevant certifications and training, and specific work location. Based on the factors above, Databricks anticipates utilizing the full width of the range. The total compensation package for this position may also include eligibility for annual performance bonus, equity, and the benefits listed above. For more information regarding which range your location is in visit our page here. Local Pay Range

$228,600 — $314,250 USD

About Databricks Databricks is the data and AI company. More than 10,000 organizations worldwide — including Comcast, Condé Nast, Grammarly, and over 50% of the Fortun 500 — rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. For specific details on the benefits offered in your region, please visit Our Commitment to Diversity and Inclusion At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Databricks are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and other protected characteristics. Compliance If access to export‑controlled technology or source code is required for performance of job duties, it is within Employer's discretion whether to apply for a U.S. government license for such positions, and Employer may decline to proceed with an applicant on this basis alone. #J-18808-Ljbffr Menlo Ventures

Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the Sr. Manager, Engineering - AI Gateway (LLM Inference) in New York, NY vacancy
  •  ...Senior Manager Of Software Engineering When you mentor and advise multiple technical teams and move financial...  ...-level software engineers delivering AI-enabled capabilities across the...  ...strategies for use-case delivery, including LLM integration approaches, evaluation... 
    Senior
    Flexible hours
    Shift work

    Chase

    New York, NY
    3 days ago
  • $229.9k - $262.4k

     ...Senior Lead AI Engineer (LLM Gateway, FM Hosting) Overview: At Capital...  ...scientists, technical program managers, and product managers to...  ...training, large language model inference, similarity search,...  ...MA: $229,900 - $262,400 for Sr. Lead AI Engineer... 
    Senior
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    6 days ago
  •  ...Senior Principal Software Engineer We're looking for a...  ...to-end model lifecycle management, including training,...  ...optimization using Model Inference servers such as Triton...  ...production operations for AI workloads, including...  ...architecting and deploying LLM & GNN solutions on AWS... 
    Senior

    Chase

    Jersey City, NJ
    3 days ago
  • BaseTen Labs, Inc. seeks an Engineering Manager (Player & Coach) to lead a team focused on LLM inference workloads. You will mentor engineers, ensuring high-quality delivery and aligning project goals with company objectives. Applicants should have a strong background... 
    Suggested
    Flexible hours

    BaseTen Labs, Inc.

    New York, NY
    6 hours ago
  • $286.2k - $326.7k

     ...Sr Director, AI Engineering Overview: At Capital One, we are creating responsible...  ..., technical program managers, and product managers to deliver...  ..., large language model inference, similarity search,...  ...introduce state-of-the-art LLM optimization techniques to... 
    Senior
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  • $174k - $189k

     ...At TradeStation, we are building an AI-First culture. We expect team members...  ...Are Looking For We are looking for a Sr. Manager, Software Engineering to lead the build‑out of TradeStation...  ...WebSockets) Strong understanding of API gateway and management platforms (e.g., Kong,... 
    Senior
    Work experience placement
    Local area
    Remote work
    Flexible hours

    TradeStation

    New York, NY
    4 days ago
  •  ...Sr Manager of Software Engineering This is your chance to change the path of your career and guide a team to...  ...Encourage responsible, effective use of AI developer tools (e.g., Copilot) to...  ...on experience with AI (e.g., Co-Pilot, LLM). ~ Experience with Java. ~ Experience... 
    Senior

    Chase

    New York, NY
    3 days ago
  • $300k

     ...interpretable, and steerable AI systems. We want AI to...  ...committed researchers, engineers, policy experts, and...  .... About The Role Our Inference team builds and...  ...production workloads. Managing multi‑region deployments...  ...traffic management systems. LLM inference optimization,... 
    Senior
    Worldwide

    Menlo Ventures

    New York, NY
    4 days ago
  • $176.17k - $251.67k

     ...Machine Learning Engineering Manager - LLM Serving (Remote - US) We are currently looking for a Machine...  ..., and driving adoption of innovative AI‑powered features. This is a highly visible...  ..., offline evaluation, and real‑time inference. Oversee the development and... 
    Remote work
    Flexible hours

    Jobgether

    New York, NY
    4 days ago
  • A leading data and AI platform company in New York is seeking a Sr. Manager of Engineering to lead teams focused on developing innovative products utilizing ML/AI techniques. The successful candidate will have a minimum of 8 years in building robust applications and will... 
    Senior

    Menlo Ventures

    New York, NY
    3 days ago
  • $300k

     ...Staff + Sr. Software Engineer, Inference San Francisco, CA | New York City, NY | Seattle, WA About...  ..., interpretable, and steerable AI systems. We want AI to be safe and...  ...balancing, request routing, or traffic management systems LLM inference optimization, batching,... 
    Senior
    Work at office
    Worldwide
    Visa sponsorship
    Flexible hours

    anthropic

    New York, NY
    3 days ago
  • $180k - $240k

    A leading AI company seeks a Senior Software Engineer to join their LLM team. This person will build and maintain the LLM gateway service that connects customers to multiple LLM providers. Responsibilities include working closely with the customer success team to understand... 
    Senior
    Remote work

    AssemblyAI

    New York, NY
    3 days ago
  • A leading financial services company is seeking a technical professional to develop AI and ML algorithms. The ideal candidate has at least 4 years of experience in programming and deploying AI solutions on cloud platforms. Responsibilities include building state-of-the... 
    Senior

    Capital One

    New York, NY
    3 days ago
  • THE ROLE As an Engineering Manager (Player & Coach), you will lead and mentor a team of Forward...  ...on building, scaling, and optimizing LLM inference workloads for Baseten customers. Applying...  ...high performance, low latency AI applications on Baseten’s platform. FDE... 
    Temporary work
    Flexible hours

    BaseTen Labs, Inc.

    New York, NY
    4 days ago
  • A leading financial services company in New York seeks an experienced AI Engineer to develop and optimize machine learning algorithms. Candidates should hold a Bachelor's or Master's degree and have significant experience programming in Python and leveraging cloud platforms... 
    Senior

    Capital One

    New York, NY
    1 day ago
  • $192.1k - $264.18k

     ...As the manager of the Digital Natives Solutions Architect (SA) team, you will focus on growing...  ...with sales, services, product, and engineering to drive solutions and outcomes for these...  ...across a wide range of data, analytical, and AI workloads. Adapt the SA team's skills and... 
    Senior
    Local area
    Worldwide

    Databricks

    New York, NY
    4 days ago
  • $150k - $200k

     ...UPCLEAR UpClear delivers a SaaS revenue management platform that is used by some of the...  ...POSITION OVERVIEW We are hiring a Sr. Manager Data Engineering & Analytics to define our data...  ...Professional experience integrating in-house AI models and API calls to foundational... 
    Senior
    Work at office

    UpClear

    New York, NY
    16 hours ago
  • $286.2k - $326.7k

     ...Sr Director, Software Engineer - Card Tech AI We are seeking a Senior Director of Software Engineering to join our...  ...I=E). In this role, you won't just manage; you will operate as a "Player-...  ...engineering ~2+ years of experience with LLM coding assistance tools such as... 
    Senior
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    8 days ago
  • $171k - $260k

     ...seeking an experienced and highly skilled engineering leader to drive the architecture,...  ...years of experience leading technologists to manage and solve complex technical items within...  ...application development. Experience with AI coding tools like GitHub Copilot, Cline,... 
    Senior

    JPMorgan Chase Bank, N.A.

    Jersey City, NJ
    2 days ago
  • $158.6k - $181k

     ...Associate, Data Scientist - LLM Customization TeamAt...  ...the Open Banking future. AI is transforming every...  ...applied researchers, software engineers, machine learning engineers and product managers to deliver AI powered...  ...in training data and inference volumes. You have experience... 
    Full time
    Part time
    Local area
    Flexible hours

    Capital One

    New York, NY
    3 days ago
  •  ...Senior Manager Of Cloud Engineering As a global leader in cybersecurity, CrowdStrike protects the people, processes and technologies that drive...  ...ve redefined modern security with the world's most advanced AI-native platform. Our customers span all industries, and they... 
    Senior
    Work at office
    2 days per week

    CrowdStrike

    New York, NY
    1 day ago
  •  ...Senior Vice President, Engineering Leader We're seeking an experienced and highly skilled...  ...years of experience leading technologists to manage and solve complex technical items within...  ...development. Experience with AI coding tools like GitHub Copilot, Cline,... 
    Senior

    Chase

    Jersey City, NJ
    3 days ago
  •  ...Staff + Sr. Software Engineer, Inference Deployment San Francisco, CA | New York City, NY | Seattle, WA...  ...reliable, interpretable, and steerable AI systems. We want AI to be safe and...  ...intersection of automation and resource management, this team will give you an outsized... 
    Senior
    Work at office
    Visa sponsorship
    Flexible hours
    Shift work

    anthropic

    New York, NY
    3 days ago
  • $180k - $240k

     ...at the forefront of Speech AI, creating powerful models for...  ...Senior Software Engineer to join our LLM team. This role is focused...  ...building and maintaining our LLM gateway service—a unified API platform...  ...load balancing, and traffic management Experience with circuit... 
    Senior
    Full time
    Remote work
    Easy work

    AssemblyAI

    New York, NY
    3 days ago
  • $176.5k - $201.4k

     ...York City seeks a Principal Associate, Data Scientist to join its LLM Customization Team. The ideal candidate is skilled in Natural...  ...has experience collaborating in cross-functional teams to develop AI-driven products. Responsibilities include leveraging advanced technologies... 

    Capital One National Association

    New York, NY
    2 days ago
  • $195k - $275k

     ...securities, investment management, and wealth management...  ...Machine learning and Gen AI Platform team(s),...  ...overall cloud adoption and engineering roadmap and ensure scalable...  ...Responsibilities: Sr Hand-On Engineer who...  ...multi-agent workflows. LLM Implementation: Deep... 
    Senior
    Temporary work
    Worldwide

    Morgan Stanley

    New York, NY
    3 days ago
  • $204k - $259k

     ...Senior Machine Learning Engineer – VLM/LLM Evaluation Waymo is an autonomous driving technology...  ...S. states. The mission of the Waymo AI Foundations team is to develop machine...  ...demonstration, generative modeling, Bayesian inference, hierarchical learning, and robust... 
    Senior
    Full time
    Temporary work
    Remote work

    Waymo

    New York, NY
    4 days ago
  • $280k - $325k

     ...SoundCloud. As a Senior Principal Engineer, you will combine deep...  ...architecting the agentic workflows and LLM-assisted systems that will...  ...helping to lead the charge on AI-native engineering Stay...  ...infrastructure and lifecycle management ~ Hands-on familiarity with... 
    Senior
    Temporary work
    Work at office
    Work from home
    Flexible hours

    SoundCloud

    New York, NY
    16 hours ago
  • $186.1k - $292.81k

     ...transforming the way companies plan, manage, and optimize sales...  ...revenue performance. About the Role AI agents are becoming central to...  .... We’re looking for a Senior Engineering Manager to own our AI agent...  ...responsible for our internal agent SDK, LLM orchestration layer, and the... 
    Senior
    Remote work
    Flexible hours
    Shift work

    CAPTIVATEIQ INC

    New York, NY
    4 days ago
  •  ...Consultants is seeking an experienced Artificial Intelligence Engineer to design and deploy AI/ML and Generative AI solutions addressing real-world...  ...to deliver AI-driven applications using production-grade LLM-powered systems. The ideal candidate has 5+ years in AI/ML... 
    Senior

    Bluetick Consultants

    New York, NY
    16 hours ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Sr. Manager, Engineering - AI Gateway (LLM Inference). Be the first to apply!