Sign up to access all features of our service.
  • Job search
  • Favorites
  • Create a CV
    New
  • Salaries
  • Subscriptions

Inference Engineer

General Compute

About us General Compute is the neocloud for alternative chips. Inference is fragmenting: purpose-built silicon from SambaNova, Cerebras, Positron, d-Matrix, and others already beats GPUs on decode, and we productionize that hardware - we buy the racks, find the data center space, and run it for our customers. Each piece of hardware runs the workload it's actually built for: prefill stays on GPUs, decode moves to the chip built for it, and today that means generating tokens 5-7× faster than existing GPU-based competitors. Our customers are frontier labs, fast-growing AI application companies, and asset-light clouds. We closed a $15M seed round in May 2026, and have since closed a $400M debt facility - $100M funded upfront by Upper90, with the balance available for drawdown - collateralized by our inference chips. About the Role Getting a model correct and fast on our silicon is only half the problem - the other half is serving it. You'll build and own the inference layer that sits between a bought-up model and a live customer request: request scheduling, batching, KV-cache management, autoscaling across our ASIC fleet, and the failure modes that only show up at real traffic and real scale. This is an early-stage role on a small team, which means the scope is wide and the ownership is real: there's no separate SRE org to hand reliability to and no platform team to hand infra to. You'll design the serving architecture, then be the person paged when it breaks. The bet is that a serving stack built specifically for our hardware - not adapted from a GPU-first framework - is a durable edge, and you're the person who proves that out in production. What You'll Do: Own the inference serving stack end-to-end. Design and build the system that takes a bring-up-verified model and serves it in production: request routing, batching, scheduling, and autoscaling a single model's serving replicas. Push cost-per-token down. Continuously tune batching strategy, KV-cache handling, and hardware utilization to widen the throughput advantage over GPU-based serving. Build for reliability from day one. Put in place the monitoring, alerting, and failover that make a fast-moving inference stack trustworthy under real customer load - and be the one who responds when it isn't. Work at the boundary with the compiler and bring-up team. Define the interface between "a model is correct and compiled" and "a model is live and fast," and push issues back to the right side of that line. Shape the roadmap, not just the backlog. As a founding engineer, you'll help decide what we build next in serving - multi-tenant isolation, speculative decoding, new scheduling strategies - not just execute a spec someone else wrote. Set the technical bar for the team you're helping build. Early architecture and code-quality decisions you make here will shape how the serving team operates as it grows. What We Need From You: 5+ years building and operating production systems at the infrastructure layer, ideally including a high-throughput or low-latency serving system. Direct experience with LLM inference serving - request batching, KV-cache management, continuous batching, or similar - in a production environment, not just research code. Comfortable owning reliability: you've been on call for a system that mattered, and you design for failure rather than reacting to it after the fact. Strong systems fundamentals - concurrency, networking, scheduling - deep enough to reason about performance at the hardware level, not just the application level. Self-directed and comfortable with ambiguity. This is a founding role: there's no existing playbook to follow, and you'll help write it. Nice-to-Haves: Experience serving models on non-NVIDIA accelerators (TPU, Trainium/Inferentia, Tenstorrent, Groq, Cerebras, or similar). Familiarity with serving frameworks such as vLLM, TGI, TensorRT-LLM, or SGLang, and an opinion on where they fall short. Experience running infrastructure at a small company or in a founding/early-engineer capacity before. Exposure to capacity planning or fleet management for specialized hardware. General Compute

Vacancy posted 2 days ago
Similar jobs that could be interesting for youBased on the Inference Engineer in New York, NY vacancy
  • $184k - $287.5k

    We're now looking for a Sr. Inference Engineer, for GPU Kernel Optimization! What does it take to push every LLM inference operation to its performance ceiling? Our LLM Inference Performance Analysis and Optimization team builds the answer from the ground up. We develop... 
    Suggested
    Full time

    Nvidia

    New York, NY
    1 day ago
  • $194k - $349k

     ...file I/O, hashing, work queues, and memory management. Every engineering team that builds sensor capabilities depends on this layer, so...  ...is building a shared platform for on-sensor machine learning inference runtime: a stable C API, backed by an isolated out-of-process... 
    Suggested
    Permanent employment
    Work experience placement
    Work at office
    Local area
    New York, NY
    25 days ago
  • $166.9k - $278.1k

     ...better working world.Role SummaryWe are seeking a Forward Deployed Engineer (FDE) - Manager to help clients solve complex business...  ...incident response.Ability to guide decisions on model selection, inference performance, latency, throughput, and cost.Software, Cloud, and... 
    Suggested
    Immediate start

    EY (Ernst & Young)

    New York, NY
    4 days ago
  • $134.7k - $202.2k

     ...initiatives, applying strong technical capability and hands-on engineering experience. Contributes to the design, development, delivery,...  ...(e.g. LoRA, QLoRA, PEFT, NeMo Framework etc.)Familiarity with inference optimization principles — latency, throughput, and cost management... 
    Suggested
    Immediate start

    EY (Ernst & Young)

    New York, NY
    4 days ago
  • $250k - $300k

     ...****@*****.***: (***) ***-****Job Title: AI Research Engineer - Deep Learning InferenceLocation: New York, NY | London,...  ...learning research and hardware acceleration, building low-latency inference systems that process continuous global data streams to drive... 
    Suggested

    Objective Paradigm

    New York, NY
    9 hours ago
  •  ...VP Hardware Engineering Location: Remote - New York, NY or San Francisco, CA A stealth-mode AI hardware startup is turning breakthrough research in low-latency large-language-model inference into purpose-built silicon. After proving the concept in software and FPGA... 
    Remote work
    Flexible hours

    Blue Signal Search

    New York, NY
    a month ago
  •  ...Job Summary We are seeking a hands-on Senior Network Test & Automation Engineer with strong expertise in enterprise networking, network testing, Python automation, Linux, and AI-assisted engineering. The candidate will be responsible for testing and certifying network... 
    Contract work

    PB consulting

    New York, NY
    4 days ago
  • $18 - $25 per hour

     ...in-depth market knowledge, the Eastern District can handle projects of any size. Kiewit Eastern District is currently seeking Engineer Interns for their Summer 2027 Intern Program! Kiewit’s Eastern District primarily performs heavy civil projects in infrastructure... 
    Full time
    Contract work
    Temporary work
    Summer work
    Internship
    Summer internship
    Work at office
    Relocation
    Weekend work

    Kiewit Corporation

    New York, NY
    1 day ago
  • $160k - $250k

     ...summary of the threats detected in their environments.This is a Sr. Engineering Manager role in the CTIO team. In this role you will build and...  ...pipelines that feed them to delivering efficient on-device inference, and you will partner closely with data science, cloud and... 
    Full time
    Work experience placement
    Work at office
    Local area
    Remote work

    CrowdStrike

    New York, NY
    1 day ago
  • Zipporin is seeking an experienced Prompt Systems Engineer to join our AI Engineering team in New York, NY on a hybrid basis. In this...  ...context compression, and token optimization strategies to minimize inference latency and operational costs.Stay abreast of generative AI... 
    Temporary work

    Zipporin

    New York, NY
    12 days ago
  • $100k - $175k

     ...Job Description Job Description Mechanical HVAC Engineer – MEP Building Systems New York City | Hybrid | Full-Time 3–15 Years of Experience | $100,000–$175,000 Base Salary + Bonus Award-Winning MEP Consulting Engineering Firm | Rewarding Environment | Professional... 
    Full time
    For contractors
    Work at office

    Agile Business Consulting

    New York, NY
    10 days ago
  • $153k - $204k

     ...without sacrificing performance.About the role:As a Senior Software Engineer on HAVOCK's Runtime & Virtualization team, you'll design and...  ...isolation, and runtime efficiency for latency-sensitive AI inference and training workloads.Extend virtualization infrastructure... 
    Full time
    Temporary work
    Casual work
    Work at office
    Flexible hours

    CoreWeave

    New York, NY
    1 hour ago
  • $184k - $287.5k

    NVIDIA is seeking an exceptional Manager, Deep Learning Inference Software, to lead a world-class engineering team advancing the state of AI model deployment. You will shape the software powering today’s most sophisticated AI systems — from large language models to multimodal... 
    Full time
    Remote work
    Worldwide

    Nvidia

    New York, NY
    2 days ago
  • $158.1k - $213.8k

     ...innovation by combining cloud scale with the world’s most talented engineers. Our team covers multiple disciplines including silicon...  ...Machine Learning chips. Inferentia delivers best-in-class ML inference performance at the lowest cost in the cloud to our AWS customers... 
    Internship
    Flexible hours

    Amazon

    New York, NY
    3 days ago
  • $207k - $275k

     ...mission-critical. We’re seeking a Senior Manager of Production Engineering to lead and expand our SRE team and practices. This leader...  ...working in AI infrastructure environments supporting training or inference at scale.Background in compliance, reliability risk modeling,... 
    Permanent employment
    Full time
    Temporary work
    Casual work
    Work at office
    Flexible hours

    CoreWeave

    New York, NY
    4 days ago
  • $100k - $115k

     ...Job Description Job Description Description: KSE is a mid-size construction management and engineering firm ranking in the ENR Top 100 for CM and ENR Top 500 for design services. With seven offices across four states (NY, NJ, PA, and CT), the firm has been providing... 
    Contract work
    Temporary work
    For contractors
    For subcontractor
    Flexible hours

    KS Engineers, P.C.

    New York, NY
    24 days ago
  • $350k

     ...Performance Engineer, Inference Systems About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed... 
    Full time
    Work at office
    Visa sponsorship
    Flexible hours

    Anthropic

    New York, NY
    8 days ago
  • $2,900 - $5,800 per month

     ...Navy’s high-tech fleet of ships, aircraft, equipment and personnel. At the center of these projects is a talented group of Civil Engineers who help to ensure that each initiative is conceived, planned and completed on time, in budget and according to specification. If... 
    Civilian Contractor
    Full time
    Contract work
    Part time
    Work at office

    U.S. Navy

    New York, NY
    9 hours ago
  • $80 - $84 per hour

     ...platform as a product. Our users are Applied Scientists and Software Engineers, and our goal is to make going from experiment to production...  ..., and generative AI use cases. The platform handles online inference and model evaluation, and it connects product teams to the... 
    Temporary work

    Braintrust

    New York, NY
    17 days ago
  • $103.71k - $138.28k

     ...ownership, deliver meaningful impact, and help shape the future of AI‑ready connectivity, join us today. The Role The Field Engineering Quality Control Engineer is responsible for overseeing and managing quality assurance processes that support Field Engineering... 
    Full time
    Temporary work
    For contractors

    Lumen

    New York, NY
    9 hours ago
  • $229.9k - $262.4k

    AI Engineer 5 (FM Hosting, LLM Inference) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has been an industry leader in using machine learning to create real-time, personalized customer experiences.... 
    Full time
    Part time
    Local area

    Capital One

    New York, NY
    1 day ago
  •  ...come to life in landmark projects around the world when you partner with Syska Hennessy Group! As a global leader in consulting, engineering, and commissioning services, we specialize in highly technical buildings and the systems that bring them to life.  We are seeking... 
    Temporary work
    Summer work
    Work at office
    Flexible hours

    Syska Hennessy Group

    New York, NY
    15 days ago
  • $100k - $130k

     ..., and functionally tested to meet the clients requirements and design intent.    About the job: ~ A commissioning project engineer reports to a team leader and is responsible for the delivery and execution of our professional services tasks for equipment in mechanical... 
    Internship
    Work at office

    FST Technical Services

    New York, NY
    6 days ago
  • $125k - $140k

    Commissioning Engineer, Boiler & HVAC SystemsLocation: NYC & Westchester County, NY (field-based, in-person)Job Type: Full-time, in-personIndustry: Commercial Mechanical Contracting, Heating Systems Installation and CommissioningPay: $125,000 - $140,000 base, plus incentive... 
    Full time
    For contractors
    Local area
    Remote work

    Cobalt Recruitment

    New York, NY
    2 days ago
  •  ...Building Automation Controls Design Engineer My client is a well established Building Automation and Energy Management company. Looking for Design Engineer with 10 or more years experience in Controls systems related to Building Automation. This person will be a degreed... 

    Liberty Personnel Services, Inc.

    New York, NY
    1 day ago
  •  ...architects, lenders, contractors, and government agencies, which go back many years. We are currently seeking a Construction Field Engineer to join our team. RESPONSIBILITIES: This position is responsible for providing technical engineering and information to... 
    Full time
    For contractors
    Work experience placement
    For subcontractor

    Perfetto Contracting Co. Inc.

    New York, NY
    3 days ago
  • $300k

     ...universities to solve problems. Our trading teams are each comprised of a dynamic group of traders, quantitative researchers, and engineers who work together to examine the global markets, seeking to understand the complexities of various traded products and exchanges.... 
    Full time
    Internship
    Visa sponsorship
    Work visa
    Flexible hours

    Jump Trading

    New York, NY
    2 days ago
  • IT Support / Operations Engineer Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling... 
    Work at office
    Flexible hours
    3 days per week

    Baseten

    New York, NY
    2 days ago
  • Baseten IT Operations Engineer Baseten powers mission-critical inference for the world's most dynamic AI companies. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting... 
    Work at office
    Remote work
    Flexible hours

    CapitalG

    New York, NY
    2 days ago
  • $130k - $190k

     ...Job Description Job Description Position: Principal Consultant – Mechanical Engineer Location: New York, New York Salary Range: $130,000 – $190,000 annually, depending on experience, education, and licensure Work Arrangement: Hybrid Employment Type... 
    Full time
    Temporary work
    For contractors
    Work at office
    Local area
    Remote work

    LaBeau Collective Talent Partners

    New York, NY
    2 days ago

Do you want to receive more vacancies?

Subscribe and receive similar vacancies to Inference Engineer. Be the first to apply!