Inference Engineer
General Compute
About us General Compute is the neocloud for alternative chips. Inference is fragmenting: purpose-built silicon from SambaNova, Cerebras, Positron, d-Matrix, and others already beats GPUs on decode, and we productionize that hardware - we buy the racks, find the data center space, and run it for our customers. Each piece of hardware runs the workload it's actually built for: prefill stays on GPUs, decode moves to the chip built for it, and today that means generating tokens 5-7× faster than existing GPU-based competitors. Our customers are frontier labs, fast-growing AI application companies, and asset-light clouds. We closed a $15M seed round in May 2026, and have since closed a $400M debt facility - $100M funded upfront by Upper90, with the balance available for drawdown - collateralized by our inference chips. About the Role Getting a model correct and fast on our silicon is only half the problem - the other half is serving it. You'll build and own the inference layer that sits between a bought-up model and a live customer request: request scheduling, batching, KV-cache management, autoscaling across our ASIC fleet, and the failure modes that only show up at real traffic and real scale. This is an early-stage role on a small team, which means the scope is wide and the ownership is real: there's no separate SRE org to hand reliability to and no platform team to hand infra to. You'll design the serving architecture, then be the person paged when it breaks. The bet is that a serving stack built specifically for our hardware - not adapted from a GPU-first framework - is a durable edge, and you're the person who proves that out in production. What You'll Do: Own the inference serving stack end-to-end. Design and build the system that takes a bring-up-verified model and serves it in production: request routing, batching, scheduling, and autoscaling a single model's serving replicas. Push cost-per-token down. Continuously tune batching strategy, KV-cache handling, and hardware utilization to widen the throughput advantage over GPU-based serving. Build for reliability from day one. Put in place the monitoring, alerting, and failover that make a fast-moving inference stack trustworthy under real customer load - and be the one who responds when it isn't. Work at the boundary with the compiler and bring-up team. Define the interface between "a model is correct and compiled" and "a model is live and fast," and push issues back to the right side of that line. Shape the roadmap, not just the backlog. As a founding engineer, you'll help decide what we build next in serving - multi-tenant isolation, speculative decoding, new scheduling strategies - not just execute a spec someone else wrote. Set the technical bar for the team you're helping build. Early architecture and code-quality decisions you make here will shape how the serving team operates as it grows. What We Need From You: 5+ years building and operating production systems at the infrastructure layer, ideally including a high-throughput or low-latency serving system. Direct experience with LLM inference serving - request batching, KV-cache management, continuous batching, or similar - in a production environment, not just research code. Comfortable owning reliability: you've been on call for a system that mattered, and you design for failure rather than reacting to it after the fact. Strong systems fundamentals - concurrency, networking, scheduling - deep enough to reason about performance at the hardware level, not just the application level. Self-directed and comfortable with ambiguity. This is a founding role: there's no existing playbook to follow, and you'll help write it. Nice-to-Haves: Experience serving models on non-NVIDIA accelerators (TPU, Trainium/Inferentia, Tenstorrent, Groq, Cerebras, or similar). Familiarity with serving frameworks such as vLLM, TGI, TensorRT-LLM, or SGLang, and an opinion on where they fall short. Experience running infrastructure at a small company or in a founding/early-engineer capacity before. Exposure to capacity planning or fleet management for specialized hardware. General Compute
$184k - $287.5k
We're now looking for a Sr. Inference Engineer, for GPU Kernel Optimization! What does it take to push every LLM inference operation to its performance ceiling? Our LLM Inference Performance Analysis and Optimization team builds the answer from the ground up. We develop...SuggestedFull time$194k - $349k
...file I/O, hashing, work queues, and memory management. Every engineering team that builds sensor capabilities depends on this layer, so... ...is building a shared platform for on-sensor machine learning inference runtime: a stable C API, backed by an isolated out-of-process...SuggestedPermanent employmentWork experience placementWork at officeLocal area$166.9k - $278.1k
...better working world.Role SummaryWe are seeking a Forward Deployed Engineer (FDE) - Manager to help clients solve complex business... ...incident response.Ability to guide decisions on model selection, inference performance, latency, throughput, and cost.Software, Cloud, and...SuggestedImmediate start$134.7k - $202.2k
...initiatives, applying strong technical capability and hands-on engineering experience. Contributes to the design, development, delivery,... ...(e.g. LoRA, QLoRA, PEFT, NeMo Framework etc.)Familiarity with inference optimization principles — latency, throughput, and cost management...SuggestedImmediate start$250k - $300k
...****@*****.***: (***) ***-****Job Title: AI Research Engineer - Deep Learning InferenceLocation: New York, NY | London,... ...learning research and hardware acceleration, building low-latency inference systems that process continuous global data streams to drive...Suggested- ...VP Hardware Engineering Location: Remote - New York, NY or San Francisco, CA A stealth-mode AI hardware startup is turning breakthrough research in low-latency large-language-model inference into purpose-built silicon. After proving the concept in software and FPGA...Remote workFlexible hours
- ...Job Summary We are seeking a hands-on Senior Network Test & Automation Engineer with strong expertise in enterprise networking, network testing, Python automation, Linux, and AI-assisted engineering. The candidate will be responsible for testing and certifying network...Contract work
$18 - $25 per hour
...in-depth market knowledge, the Eastern District can handle projects of any size. Kiewit Eastern District is currently seeking Engineer Interns for their Summer 2027 Intern Program! Kiewit’s Eastern District primarily performs heavy civil projects in infrastructure...Full timeContract workTemporary workSummer workInternshipSummer internshipWork at officeRelocationWeekend work$160k - $250k
...summary of the threats detected in their environments.This is a Sr. Engineering Manager role in the CTIO team. In this role you will build and... ...pipelines that feed them to delivering efficient on-device inference, and you will partner closely with data science, cloud and...Full timeWork experience placementWork at officeLocal areaRemote work- Zipporin is seeking an experienced Prompt Systems Engineer to join our AI Engineering team in New York, NY on a hybrid basis. In this... ...context compression, and token optimization strategies to minimize inference latency and operational costs.Stay abreast of generative AI...Temporary work
$100k - $175k
...Job Description Job Description Mechanical HVAC Engineer – MEP Building Systems New York City | Hybrid | Full-Time 3–15 Years of Experience | $100,000–$175,000 Base Salary + Bonus Award-Winning MEP Consulting Engineering Firm | Rewarding Environment | Professional...Full timeFor contractorsWork at office$153k - $204k
...without sacrificing performance.About the role:As a Senior Software Engineer on HAVOCK's Runtime & Virtualization team, you'll design and... ...isolation, and runtime efficiency for latency-sensitive AI inference and training workloads.Extend virtualization infrastructure...Full timeTemporary workCasual workWork at officeFlexible hours$184k - $287.5k
NVIDIA is seeking an exceptional Manager, Deep Learning Inference Software, to lead a world-class engineering team advancing the state of AI model deployment. You will shape the software powering today’s most sophisticated AI systems — from large language models to multimodal...Full timeRemote workWorldwide$158.1k - $213.8k
...innovation by combining cloud scale with the world’s most talented engineers. Our team covers multiple disciplines including silicon... ...Machine Learning chips. Inferentia delivers best-in-class ML inference performance at the lowest cost in the cloud to our AWS customers...InternshipFlexible hours$207k - $275k
...mission-critical. We’re seeking a Senior Manager of Production Engineering to lead and expand our SRE team and practices. This leader... ...working in AI infrastructure environments supporting training or inference at scale.Background in compliance, reliability risk modeling,...Permanent employmentFull timeTemporary workCasual workWork at officeFlexible hours$100k - $115k
...Job Description Job Description Description: KSE is a mid-size construction management and engineering firm ranking in the ENR Top 100 for CM and ENR Top 500 for design services. With seven offices across four states (NY, NJ, PA, and CT), the firm has been providing...Contract workTemporary workFor contractorsFor subcontractorFlexible hours$350k
...Performance Engineer, Inference Systems About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed...Full timeWork at officeVisa sponsorshipFlexible hours$2,900 - $5,800 per month
...Navy’s high-tech fleet of ships, aircraft, equipment and personnel. At the center of these projects is a talented group of Civil Engineers who help to ensure that each initiative is conceived, planned and completed on time, in budget and according to specification. If...Civilian ContractorFull timeContract workPart timeWork at office$80 - $84 per hour
...platform as a product. Our users are Applied Scientists and Software Engineers, and our goal is to make going from experiment to production... ..., and generative AI use cases. The platform handles online inference and model evaluation, and it connects product teams to the...Temporary work$103.71k - $138.28k
...ownership, deliver meaningful impact, and help shape the future of AI‑ready connectivity, join us today. The Role The Field Engineering Quality Control Engineer is responsible for overseeing and managing quality assurance processes that support Field Engineering...Full timeTemporary workFor contractors$229.9k - $262.4k
AI Engineer 5 (FM Hosting, LLM Inference) At Capital One, we are creating responsible and reliable AI systems, changing banking for good. For years, Capital One has been an industry leader in using machine learning to create real-time, personalized customer experiences....Full timePart timeLocal area- ...come to life in landmark projects around the world when you partner with Syska Hennessy Group! As a global leader in consulting, engineering, and commissioning services, we specialize in highly technical buildings and the systems that bring them to life. We are seeking...Temporary workSummer workWork at officeFlexible hours
$100k - $130k
..., and functionally tested to meet the clients requirements and design intent. About the job: ~ A commissioning project engineer reports to a team leader and is responsible for the delivery and execution of our professional services tasks for equipment in mechanical...InternshipWork at office$125k - $140k
Commissioning Engineer, Boiler & HVAC SystemsLocation: NYC & Westchester County, NY (field-based, in-person)Job Type: Full-time, in-personIndustry: Commercial Mechanical Contracting, Heating Systems Installation and CommissioningPay: $125,000 - $140,000 base, plus incentive...Full timeFor contractorsLocal areaRemote work- ...Building Automation Controls Design Engineer My client is a well established Building Automation and Energy Management company. Looking for Design Engineer with 10 or more years experience in Controls systems related to Building Automation. This person will be a degreed...
- ...architects, lenders, contractors, and government agencies, which go back many years. We are currently seeking a Construction Field Engineer to join our team. RESPONSIBILITIES: This position is responsible for providing technical engineering and information to...Full timeFor contractorsWork experience placementFor subcontractor
$300k
...universities to solve problems. Our trading teams are each comprised of a dynamic group of traders, quantitative researchers, and engineers who work together to examine the global markets, seeking to understand the complexities of various traded products and exchanges....Full timeInternshipVisa sponsorshipWork visaFlexible hours- IT Support / Operations Engineer Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling...Work at officeFlexible hours3 days per week
- Baseten IT Operations Engineer Baseten powers mission-critical inference for the world's most dynamic AI companies. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting...Work at officeRemote workFlexible hours
$130k - $190k
...Job Description Job Description Position: Principal Consultant – Mechanical Engineer Location: New York, New York Salary Range: $130,000 – $190,000 annually, depending on experience, education, and licensure Work Arrangement: Hybrid Employment Type...Full timeTemporary workFor contractorsWork at officeLocal areaRemote work
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to Inference Engineer. Be the first to apply!











